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Turning On to LightSwitch W3 


M icrosoft once owned the novice developer space 
with its Visual Basic (VB) 6 product. VB6 was 
easy to learn and was an incredibly productive devel- 
opment environment. However, VB6 was no match for 
“real” languages like C++ and Java. Microsoft needed 
to move to the more enterprise-capable .NET platform 
to effectively compete against Java in the enterprise. 

When Microsoft took a turn down the .NET road, 
VB6 with its simplicity and productivity was lost for 
good. Microsoft’s newer languages like C# and the 
.NET-based VB (such as VB 2010) are definitely more 
capable and robust than VB6 ever was. However, 
they’re also vastly more complex. That low initial 
learning hurdle that VB6 users enjoyed isn’t part of 
the .NET equation. 


Easy to Learn Development: 

Still MIA 

Microsoft has been looking for a way to get back 
into the novice developer space for some time, but 
so far they haven’t gotten it quite right. First, they 
tried with the failed Web Matrix product, next with 
the capable, cost effective, but still complex Visual 
Studio Express line of products, and more recently 
with yet another Web Matrix tool designed to attract 
open-source developers. Although all of these tools 
can be used for application development, they still 
require someone with developer skills and mental- 
ity. Administrators and power users aren't going 
to pick up these tools unless they’re under duress. 
That old backup plan where you can just pick up 
VB6 and program something in a couple of days is 
still MIA. 


LightSwitch: Powered by Wizards 
This is where the new Visual Studio LightSwitch 
product comes in. LightSwitch (currently in beta) is 
a wizard-driven .NET application generator along the 
lines of Iron Speed Designer. LightSwitch can create 
Windows-, web-, and Azure-based cloud applications. 
LightSwitch can generate either VB or C# applica- 
tions. It’s very data centric and can attach to a number 
of data sources, including SQL Server, SharePoint, 
Access, and SQL Azure. 

Although LightSwitch is aimed at the same target 
audience of non-developer business people as VB6 
was, the LightSwitch approach is entirely different. 
VB6’s drag-and-drop designer is gone. It’s replaced 
by LightSwitch’s wizards and templates. 

LightSwitch doesn’t have a visual design envi- 
ronment. In LightSwitch you don’t even see a 
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screen until it’s rendered. You begin creating 
LightSwitch applications by connecting to a data 
source and then selecting an existing table or 
choosing to create a new table. Next, you select 
the components you want to use. LightSwitch will 
generate the required application pages based on its 
built-in application templates. The pages or forms 
generated will be complete with all the appropriate 
basic data validations for the controls and data that 
you selected. 


Ability to Customize LightSwitch 
Apps: Still Limited in Beta 

The bottom line on LightSwitch is that it really does 
allow non-developers to create database applications. 
However, in this early beta the ability to customize 
those applications is limited, which means that the 
apps that you wind up with might not work and act 


LightSwitch really does allow 
non-developers to create 
database applications. 


the way you want. Microsoft’s early marketing touts 
LightSwitch as a tool for developing professional- 
quality applications, but that’s not really the case just 
now. The limitations in screen customization separate 
LightSwitch applications from professional applica- 
tions. However, it’s still early in the development 
cycle, and it’s certain that LightSwitch will continue 
to improve before its final release. 

It remains to be seen whether the new LightSwitch 
tool will be able to fill in the gap at the administra- 
tor, part-time developer, and beginning developer 
level, but at least it shows that Microsoft knows that 
businesses have a pain point here that needs to be 
addressed. 

At the time of this writing, pricing isn’t available. 
I expect LightSwitch will be a relatively low cost 
tool, but I don’t expect it to be a free tool. You can 
download the beta version of Visual Studio Light- 
Switch from www.microsoft.com/visualstudio/en-us/ 
lightswitch. 

Have you tried LightSwitch? Send me your 
thoughts at motey@sqlmag.com. Let’s start a con- 
versation. Also, look for more coverage in my blog 
on www.sqimag.com. SQL 

InstantDoc ID 126007 
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Intelligent Transaction 


Log Backups 


esigning a backup strategy for transaction 
D- is simple in concept. However, various 
scenarios can cause the design to be more 
complex than anticipated. For example, what if you 
have a mirrored database that automatically fails 
over as the principal database? Does your transac- 
tion log backup job start automatically? Does the 
job stop when the principal database becomes the 
mirror? What if you change the recovery model of a 
database? Will your transaction log backup job adjust 
automatically? 

I have developed a simple solution that works on 
SQL Server 2008 and SQL Server 2005. It uses what 
I call the admin/worker job concept. The admin job 
is the only job that’s scheduled. By default, it’s sched- 
uled to run every 15 minutes. (You can change that 
default if desired.) The admin job creates/edits and 
starts the worker job. (The worker job is never manu- 
ally created or scheduled to run.) The worker job is 
updated every time the admin job executes. 

The combination of the admin job and worker 
job lets you use this solution on any box and never 
have to make any modifications. It will intelligently 
execute as needed, no matter whether the database is a 
principal or mirror in Simple or Full Recovery mode. 
The admin job will identify the state of the database 
and act accordingly. 

The mnt_BackupLog stored procedure provides 
the “intelligence” in the admin job. It locates candi- 
date databases—that is, online databases that are in 
recovery mode. When candidate databases are pres- 
ent, mnt_BackupLog creates and executes a worker 
job that will back up the transaction logs in those 
databases. (Database snapshots and databases that 
are in single-user or standby mode aren’t considered 
candidate databases. Also, any databases that are part 
of log shipping are excluded.) 

Let’s take a detailed look at the admin and worker 
jobs. Then, I'll explain how to create the admin job 
using the SDM_Logs_QuickSetup.sql script. 


The Admin Job 

The admin job is a SQL Server Agent job named 
Server Daily Maintenance - Logs. Figure | outlines 
the job steps, the first of which is creating the msdb 
.dbo.SDM_Cleanup table if it doesn’t exist. This 
table stores records on transaction log backups that 
have been performed. Records are added to the 
SDM_Cleanup table by the worker job. 
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The second job step calls the mnt_CleanUp 
stored procedure into action. This stored procedure 
determines which transaction-log backup files should 
be deleted using the records in the SDM_Cleanup 
table, then deletes them. It also deletes the associ- 
ated records in SDM_Cleanup. If a backup file has 
already been deleted, mnt_CleanUp still removes the 
associated record. 

The last job step locates the candidate databases, 
then uses the mnt_BackupLog stored procedure to 
create a worker job that will back up the transaction 
logs for those databases. 


The Worker Job 
The worker job is a SQL Server Agent job named 
Maintenance_BackupLog. It will contain one step for 
each candidate database. So, if there are three candi- 
date databases, there will be three job steps. Figure 2 
shows a sample job step. 

Here’s what happens in each job step. The worker 
job first checks to see whether a full backup of the 
database exists. If not, it will perform one. It 
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MORE on the WEB 
then backs up the transaction log using either the ¢ ) Download the code at 
SQL Server native format or LiteSpeed format. InstantDoc ID 125991. 


LiteSpeed is a third-party utility from Quest 
Software. The default backup format is SQL Server 
native, so LiteSpeed isn’t required for this solution. 

After the transaction log is backed up, the worker 
job writes the backup’s fully qualified filename, type 
of backup, and database name to the SDM_Cleanup 
table. 


Getting Started 

I created SDM_Logs QuickSetup.sql to quickly set 

up this solution. (You can download this script by 

going to www.sqlmag.com, entering 125991 in the 

InstantDoc ID text box, clicking Go, then clicking 

the Download the Code Here button.) When you run 

SDM_Logs_QuickSetup.sq], it 

e Creates the mnt_BackupLog stored procedure 

e Creates the mnt_CleanUp stored procedure 

e Creates the Server Daily Maintenance - Logs 
admin job. 


Note that SDM_Logs_QuickSetup.sql and the 
two stored procedures use xp_cmdshell when they 
run. I put xp_cmdshell inside a wrapper. That way, 
if xp_cmdshell is disabled, the script and stored pro- 
cedures will enable it for a brief moment so they can 
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perform their tasks, then return xp_cmdshell to the 
disabled state. If company policy forbids you to use 
xp_cmdshell even when it’s in a wrapper, you won't be 


able to use this solution. 


Step Name Command 
Create Table IF OBJECT_ID(N'msdb.dbo.SDM_Cleanup’, N'U') IS NULL 
msdb,dbo.SDM._Cleanup, BEGIN 


CREATE TABLE msdb.dbo.SDM_Cleanup ( 

DBName sysname NOT NULL. 

FullFileName nvarchar(125) NULL. 

CreateDate datetime PRIMARY KEY NOT NULL DEFAULT 
GETDATE() 

BackupType nchar(3) 


END 


Clean Up EXEC msdb dbo .mnt_CleanUp 
@daysold=1 


Start Maintenance Backuplog | SET NOCOUNT ON 


— If job is still running from last run, skip this step. 
— Allow execution on next run 


IF ( 
SELECT TOP 1 


CASE 
WHEN sa start_execution_date IS NULL THEN 0 — Not running 
WHEN sa start_execution_date |S NOT NULL AND 
sa.stop_execution_date |S NULL THEN 1 -— Running 
WHEN sa start_execution_date IS NOT NULL AND 
sa.stop_execution_date |S NOT NULL THEN 0 — Not running 
END N’RunStatus’ 


FROM msdb dbo sysjobactivity sa WITH (NOLOCK’ 
JOIN msdb dbo. sysjobs sj WITH (NOLOCK) 
ON sa job_id = sj job_id 
WHERE sj.[name] = N’Maintenance_BackupLog’ 
ORDER BY sa Session_id DESC 
)=1 
BEGIN 
RETURN 
END 
ELSE BEGIN 
EXEC msdb.dbo mnt_BackupLog 
@Format = 'NV' 
@dir = N'E:\MSSQL\TRAN’ 
WAITFOR DELAY 00:00:09" 
IF EXISTS (SELECT * FROM msdb dbo. sysjobs 
WHERE [name] = N'Maintenance_BackupLog’ 
BEGIN 
EXEC msdb.dbo.sp_start_job 
@job_name = N'Maintenance_BackupLog’ 
END 
END 


Figure | 
Outline of the job steps in the admin job 
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Figure 2 


Sample job step from a worker job 


After you run SDM_Logs_QuickSetup.sql, you 
can change some settings for the Server Daily Main- 
tenance - Logs admin job if desired: 

e By default, the Server Daily Maintenance - 
Logs job creates the SDM_Cleanup table in 
the msdb database. I like using that database 
because I know it’s on every server and its data 
is groomed regularly. If you prefer to create it 
in a different database, you can change each 
instance of msdb.dbo.SDM_Cleanup to reflect 
the desired database in the first job step as well 
as in the mnt_BackupLog and mnt_CleanUp 
stored procedures that were created. 

e By default, the Server Daily Maintenance - 
Logs job is scheduled to run every 15 minutes. 
You can change the desired frequency in the 
New Job Schedule page. 

e By default, the mnt_CleanUp stored proce- 
dure is set to delete transaction log backup 
files that are older than 1 day. If you want to 
change that default, find the code 


EXEC msdb.dbo.mnt_CleanUp 
@DaysOld = 


in the second job step and replace 1 with an 
integer that represents the number of days you 
want the transaction log backup files to be 
kept on the local server. 

e By default, the mnt_BackupLog stored proce- 
dure has the worker job back up the transac- 
tion logs using the SQL Server native format 
and store the backup files in E:\MSSQL\ 
TRAN. If you want to change these defaults, 
find the code 


EXEC msdb.dbo.mnt_BackupLog 
@Format = N''NV'', 
@dir = N''E:\MSSQL\TRAN'' 


in the third job step. If you want to use the 
LiteSpeed format, change NV to LS in the 
@Format argument. If you want to store the 
backup files in a different location, replace 
E:\MSSQL\TRAN with the desired directory 
in the @dir argument. The directory you 
specify must exist. The worker job will create 
subdirectories in that directory as needed, 

so the SQL Server service account must have 
write permissions to that directory. 


Just Set and Forget 

After you have the admin job as you like it, you're 

all set and you can forget about it. The Server Daily 

Maintenance - Logs job will have your transaction log 

backups covered. SQL 
InstantDoc ID 125991 
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DVERTISING SUPPLEMENT TO SQL SERVER MAGAZINE 


icrosoft’s SQL Server is an 

enterprise-caliber relational 

database management system 

(RDBMS) that has proven its abil- 
ity to meet a variety of mission critical business 
needs. However, knowing how to optimally 
configure and manage SQL Server to meet 
business needs today can be tricky—especially 
as continued demand for decreased costs is 
contrasted against emerging trends and patterns 
that continue to reshape the IT landscape. 


Addressing Challenges in Data 
Management Today 

Intelligent data management today revolves 
around ensuring proper data availability, scal- 
ability, and manageability while maintaining 
cost-effectiveness. This, in turn, requires an 
understanding of how different deployment 
topologies and SQL Server workload character- 
istics can best be handled by IT infrastructure to 
ensure that emerging trends in business and IT 
don't undermine availability and manageability 
needs or negatively impact total costs of owner- 
ship for mission-critical systems. 


Trends in Business Intelligence 

For example, a perfect illustration of how busi- 
ness trends impact IT considerations is the 
continued trend toward increased adoption 
and use of Business Intelligence (BI) solutions. 
As Microsoft continues to help democratize the 
adoption of BI initiatives, and as more and more 
businesses implement BI solutions, increased 
adoption causes a “snowball effect” where more 
and more businesses implement BI solutions to 
keep up with their competition. 

This, in turn, pushes additional demand for IT 
services and infrastructure and drives up require- 
ments for additional processing power and stor- 
age capacity. 


Trends toward Higher Availability 
Another industry trend is the move toward a 
more proactive approach to systems manage- 
ment by IT departments that are seeking to be- 
come more agile. By acknowledging that system 
failures are not a question of “if; but of “when,” 
disaster recovery solutions are increasingly being 
bolstered by High Availability (HA) solutions that 
proactively account for potential outages at the 
system, hardware, or site level. 

In turn, this strategic approach to mitigating 
the costs and impact of disasters and outages 
translates into an increased demand for more 
processing and storage capacity—along with 
the need for sophisticated failover management 
solutions and capabilities. 


Architectural Considerations 

In addition to trends within IT and business, 
architectural choices (which are commonly 
driven by trends within IT) can also have broad, 
long-lasting implications for mission-critical SQL 
Server deployments. 

For example, as single systems have increas- 
ingly been replaced by highly consolidated, 
virtual systems in an effort to decrease costs 
and increase agility, continued demand for 
increased scalability and fault-tolerance is now 
starting to drive another shift, or trend, toward 
utilization of systems within public or private 
clouds. But many database workloads exhibit 
performance and security characteristics that 
aren't amenable to either consolidation or 
deployment within the cloud. 

Therefore, trying to address how trends shape 
architectural developments while deciphering 
how architectural choices impact the ability to 
respond to trends can often lead to a quandary 
where IT professionals are left to second-guess 
which trends and architectural approaches 
to heed or ignore. Happily, keeping things 
simple—and focused on cost—can have a ben- 
eficial impact on determining where to direct 
efforts and energy. 


Ensuring Solution Affordability 
Despite the complexity associated with priori- 
tizing architectural considerations and trends, 
there are a few constants within IT. For ex- 
ample, scalability, availability, and performance 
will always be in demand by end users and 
management—just as continual demand for 
decreased costs will always be a requirement 
from management. And while the constancy 
of these two competing needs may seem like 
a problem, purchasing and leveraging tech- 
nologies and solutions that acknowledge and 
address this reality provide some of the best, 
long-term benefits for IT departments. 

Happily, SQL Server shines in this regard 
due to its high degree of flexibility in meeting 
business needs by simultaneously bundling 
solid OLTP performance and scalability with a 
plethora of BI tools and solutions into the same 
license and solution—allowing IT professionals 
to offer businesses more options and capabili- 
ties for less. As affordable and versatile as SQL 
Server is, understanding SQL Server's depen- 
dence upon storage infrastructure is a crucial 
component of increasing ROI and ensuring 
continued IT success. Consequently, coupling 
SQL Server with an efficient and cost-effective 
infrastructure helps organizations realize cur- 
rent and future objectives and requirements. 


Do You Know Wh 


EMC is 


Your Partner of Choice for 
Microsoft SQL Server? 


You’re looking for a partner that under- 

stands both the Microsoft SQL Server 
platform and the information infrastructure 
that supports it. 
EMC® provides deep knowledge and best 
practices developed over almost a decade of 
experience with Microsoft SQL Server environ- 
ments, including both OLTP and BI/DW con- 
figurations. We also bring to the table thought 
leaders with elite Microsoft certifications, a 
broad range of expert services, and a compre- 
hensive Information infrastructure portfolio— 
all supported by a comprehensive alliance with 
Microsoft. 


You seek a consultative approach to de- 

signing your next-generation deployments 
of Microsoft SQL Server and other key business 
applications. 
EMC provides a complete portfolio of strategic 
consultation, planning, delivery, and support 
services across the entire lifecycle of your 
Microsoft SQL Server initiatives. Our consultants 
bring to bear a unique mix of industry, business, 
and technology expertise to solve your toughest 
challenges—addressing Business Intelligence, 
Data management, and OLTP requirements. We 
leverage customer experience, lab-validated 
EMC Proven™ Solutions, proven methodolo- 
gies, best practices, and industry standards to 
minimize risk and increase efficiency across 
your SQL Server lifecycle. 


Your business demands a higher 

level of efficiency in its information 
infrastructure. 
EMC solutions and technologies enable con- 
solidation and simplified deployments, along 
with cost-effective management and tiered 
storage efficiencies for the range of user data 
types and extreme data volumes that Microsoft 
SQL Server environments now support. With 
industry leading technologies such as Virtual 
LUN and thin provisioning coupled with the 
broadest range of platform choices, EMC can 
help you address requirements for total cost of 
ownership, scalability, optimal utilization, and 
work load balancing. You need to ensure that 
the solution you deploy can meet your require- 
ments for initial acquisition cost but also the 
cost of uptime, the cost of maintenance, and 
the cost of ongoing management. EMC delivers 
this via our leadership technologies for storage 
efficiency, including EMC VPLEX™ and Tiered 
Storage with table partitioning and fully auto- 
mated storage tiering. 


You want to leverage virtualization to 

lower TCO, improve agility, deliver flexibil- 
ity, and enable Private Clouds. 
EMC is a leading Microsoft consultancy with 
storage technology integration across Micro- 
soft’s product portfolio, and deep experience in 
information infrastructure for virtual environ- 
ments supporting both VMware® and Microsoft 
Hyper-V. EMC has the technologies, solutions, 
and expertise to help you fully leverage the 
benefits of virtualization for SQL server consoli- 
dation and improved utilization. Additionally, 
EMC offers you choice and confidence on your 
Journey to the Private Cloud by offering EMC 
Proven Solutions, engineering integration, and 
joint best practices that support both VMware 
and Microsoft virtualization technologies. 
Deploying virtualized Microsoft applications 
with an EMC storage infrastructure can create 
the efficient, protected, and easy-to-manage 
environments you need to address current and 
future requirements. 


You need to ensure data protection with 

robust backup, recovery, and restore 
solutions. 
EMC has a complete offering for backup, re- 
covery, and restore that includes tiered service 
levels for performance and recovery, disk-based 
recovery options, integration with point-in-time 
replication, and advanced software solutions 
for recovery management. These advanced 
tools offer LAN-free data protection solutions 
that are tightly integrated with Microsoft ap- 
plications and Microsoft SQL Server software, 
including support for multi-site clustering using 
storage-based replication (cluster enabler) and 
disk-based replication for backup acceleration, 
reporting, and analysis services optimization. 


You must ensure business continuity. 

EMC has deep experience across diverse 
customer environments in high-availability and 
business-continuance solutions for Microsoft 
SQL Server environments, combining hardware 
platform reliability with advanced replication 
software capabilities. And EMC’s local and 
long-distance replication solutions are validated 
and supported by reference architectures 
and proven best practices. EMC Solutions and 
technologies extend high availability across 
multiple sites with near-zero downtime with 
cluster enabler integration. EMC solutions also 
ensure continuous data protection and multi- 
site SQL-aware bookmarks for any point-in-time 
recovery with EMC RecoverPoint CDP. 


You want to partner with an industry leader in 

Microsoft SQL Server deployments. 
A Microsoft Gold Certified and Global Alliance Part- 
ner with 15 Microsoft competencies and 20 “Partner 
of the Year” awards, EMC works with Microsoft to 
develop best practices that combine software, hard- 
ware, and services to help you streamline deploy- 
ment, migration, and management of your Microsoft 
platform. Together, we help deliver the higher levels 
of information protection and access your business 
needs while increasing efficiency and lowering risk. 


To reduce risk, you want a partner with vali- 

dated best practices and technology solutions 
to ensure deployment, upgrade, and migration 
initiatives are delivered in a predictable manner. 
To ensure EMC products perform at the highest levels, 
we conduct the industry’s most comprehensive in- 
teroperability testing. We’re committed to developing 
and testing hardware and software products that are 
fully qualified with Microsoft technologies, supporting 
Microsoft users with compatible, integrated solutions. 


Best Practices for SQL Server 
Management 

By adhering to best practices for leveraging SQL 
Server's native capabilities and extending them 
where needed with third-party infrastructure and 
solutions, IT organizations can ensure that informa- 
tion workers and customers have secured access to 
the data they need while keeping costs down. 


Management Considerations 

One key way to decrease IT costs is to cut down on 
the amount of management needed—by ensuring 
that management tasks are automated and scal- 
able wherever possible. SQL Server provides excel- 
lent support in this area by virtue of its support for 
Centralized Management Server capabilities and its 
support for Policy-Based Management that enables 
proactive management. 

However, SQL Server doesn’t run in a vacuum, and 
is heavily dependent upon optimally configured IO 
subsystems in order to constantly balance the need for 
both high-performance IO against long-term storage 
needs for largely static data. Many hours are spent 
troubleshooting disk bottlenecks within SQL Server 
database environments and configuring storage has 
been challenging in the past. Therefore, simplifying 
and automating management of SQL Server's underly- 
ing storage infrastructure can provide substantial cost 
savings. EMC's FAST (Fully Automated Storage Tiering) 
is built on top of simple storage pools and can intel- 
ligently shift “hot” database workloads to high-perfor- 
mance, low-latency Flash drives and “cold” workloads 
to high-capacity, low-cost drives. You can see up to a 
38 percent reduction in acquisition cost and up to a 45 
percent reduction in power and cooling costs com- 
pared to an all Fibre Channel deployment. 


You need to accelerate the business value of 
Microsoft SQL Server. 
Whether you’re deploying, upgrading, or migrat- 
ing to Microsoft SQL Server you can acceler- 
ate the entire process and ensure exceptional 
results with EMC Solutions for SQL Server. With 
the enterprise-grade capabilities of Microsoft 
SQL Server and a powerful EMC storage infra- 
structure, you can reduce complexity and enable 
advanced consolidation throughout your entire 
organization. 


1 You want to work with a partner that has 
global reach to support your Microsoft 
SQL Server environments. 
EMC’s Global Services organization has thousands 
of consultants with deep expertise to provide a 
broad portfolio of strategic consultation, plan- 
ning, delivery, and support across the entire IT 
lifecycle, from envisioning through day-to-day 
operations. 


CROSS REFERENCE: 

For more information about EMC's FAST (Full 

Automated Storage Tiering) Solutions, visit www 

.emc.com/products/launch/fast/. 
Storage and Provisioning 
Considerations 
For mission-critical SQL Server deployments, storage 
is an essential resource. Without properly provisioned 
and optimized storage, SQL Server cannot attain basic 
needs for stability, performance, scalability, and man- 
ageability. And the key to ensuring that SQL Server 
has access to the kinds of storage it needs, is to un- 
derstand that SQL Server workloads and performance 
characteristics are typically much different from the 
IO signatures of file and applications servers. 

Likewise, it’s important to understand that even 
different kinds of SQL Server workloads have 
drastically different IO signatures and characteris- 
tics. For example, OLTP solutions trend towards a 
heavier usage of scatter-gather IO and tend to be 
more write-intensive than OLAP solutions—which 
are commonly characterized as being processor 
intensive yet subject to needing large quantities of 
sequentially accessed data. 

Consequently, a critical best practice for mission- 
critical SQL Server environments is to ensure that 
SAN administrators are familiar with SQL Server's spe- 
cialized needs and making sure that SAN administra- 
tors have the proper infrastructure and tools needed 
to address those needs. In many cases, this means 
that SAN administrators will have to accord SQL 
Server with both high-performance storage for vola- 
tile data, and large amounts of less-performant, com- 
modity storage tasked with handling less-volatile, 
and infrequently accessed data needed for auditing, 
historical trending, or regulatory compliance. 


It's also important to stress that these consid- 
erations don't imply any kind of incompatibility 
between SQL Server and Storage Array Networks. 

To the contrary, SQL Server performs best when 
deployed on highly optimized SANS. For example, 
Microsoft's Scalable Shared Databases facilitate pow- 
erful scale-out capabilities—but only via a properly 
configured SAN infrastructure such as those archi- 
tected and delivered by EMC. 


CROSS REFERENCE: 

For more information about leveraging best 
practices to architect and properly configure high- 
performance SAN infrastructure for SQL Server 
workloads, visit www.emc.com/SQLServer. 


Another best practice or consideration for optimiz- 

ing SQL Server storage requirements is to clearly 
distinguish between SQL Server's core libraries and 
execution environment (i.e., operating system and SQL 
Server binaries) and the workloads—or databases— 
being handled. While this might seem obvious to many 
IT professionals, the fact that database workloads and 
SQL Server hosts have vastly different storage require- 
ments needs to be clearly communicated to SAN and 
storage administrators in order to maximize perfor- 
mance and costs benefits. By making this distinction, IT 
departments can more easily balance HA and load- 
balancing requirements for SQL Server Instances, while 
ensuring the performance and capacity requirements 
needed to properly handle SQL Server workloads—or 
databases. Under this paradigm, IT departments can 
employ consolidation and virtualization best practices 
more readily when working with SQL Server in order to 
better increase IT agility and cut costs. 

Of course, managing such disparate and highly 
isolated workloads requires powerful SAN capabili- 
ties and tools. It also requires optimized consolidation 
solutions for easier provisioning and manageability. As 
noted previously, EMC’s FAST automates the move- 
ment and placement of data within storage systems in 
order to decrease the need for repetitive administra- 
tive tasks—which can be heavily leveraged to meet 
SQL Server data performance and retention require- 
ments. But EMC also provides highly specialized vir- 
tualization and consolidation management tools and 
solutions based on industry leading storage platforms 
such as EMC CLARIION® and EMC Symmetrix®°—which 
lower TCO by reducing over-allocation of images and 
reducing obstacles to growth. Moreover, by address- 
ing the need to treat database workloads differently 
from virtualized SQL Server hosts, IT organizations can 
utilize virtual provisioning tools to improve storage 
density and efficiency. Specifically, with intelligent 
provisioning solutions, administrators can decrease 
storage requirements and achieve better consolida- 
tion ratios by allocating space only as necessary—in 
order to reduce storage costs while maintaining 
performance requirements. 


CROSS REFERENCE: 

For more information about EMCS Virtual 
Provisioning Solutions (EMC Unified Storage and 
EMC Symmetrix), visit www.emc.com/products/ 
detail/software/symmetrix-virtual-provisioning.htm. 
or 
www.emc.com/products/detail/software/clariion- 
virtual-provisioning.htm. 


High-Availability Considerations 

When it comes to meeting business continuity re- 
quirements for SQL Server systems, primary consider- 
ations for implementing successful solutions typically 
revolve around recovery time objectives—or decreas- 
ing the amount of time (and effort) required to failover 
from one server (or site) to another. As such, regular 
testing and validation of implemented solutions is a 
critical best practice that can help facilitate success. 

Likewise, when accounting for potential failures, 
organizations with geographically distributed sys- 
tems can't afford to consider failures at merely the 
system or hardware level. They need to account for 
potential failures at the site level as well. 

Consequently, while Microsoft's Clustering Services 
provide rapid and automatic failover of mission-critical 
resources at the system level, another best practice 
many organizations need to address is Log Shipping— 
which can help account for disasters or outages at the 
site level. However, a downside to Log Shipping is that 
it does not provide automated failover. 

As such, many organizations are increasingly 
implementing solutions known as “stretch clusters”— 
which use clustering technologies to span mission- 
critical resources across multiple sites. And one great 
example of how to set up “stretch clusters” is EMC's 
RecoverPoint/Cluster Enabler—a solution that inte- 
grates with Microsoft's Clustering Services with stor- 
age array replication technologies to create failover 
clusters that extend protection to the site level—by 
spanning multiple sites. With solutions like EMC's 
Cluster Enabler, IT organizations can seamlessly inte- 
grate commonly used features like Windows Failover 
Clustering across multiple sites. This approach can 
ease operational overhead in the event of a site failure 
by providing automate failover to ensure that mission- 
critical resources are brought online in accordance 
with Recovery Time Objective (RTO) requirements. 


CROSS REFERENCE: 

For more information about EMC's Cluster Enabler, 
visit www.emc.com/collateral/software/white- 
papers/h5936-recoverpoint-cluster-enabler-wp 
.pdfPractical Application. 


When juggling SQL Server's capacity for meeting 
real-world business needs against different deploy- 
ment topologies (dedicated, clustered, or cloud) 
along with requirements for manageability, perfor- 
mance, and budgetary constraints, few organizations 


icrosoft’s SQL Server is an 

enterprise-caliber relational 

database management system 

(RDBMS) that has proven its abil- 
ity to meet a variety of mission critical business 
needs. However, knowing how to optimally 
configure and manage SQL Server to meet 
business needs today can be tricky—especially 
as continued demand for decreased costs is 
contrasted against emerging trends and patterns 
that continue to reshape the IT landscape. 


Addressing Challenges in Data 
Management Today 

Intelligent data management today revolves 
around ensuring proper data availability, scal- 
ability, and manageability while maintaining 
cost-effectiveness. This, in turn, requires an 
understanding of how different deployment 
topologies and SQL Server workload character- 
istics can best be handled by IT infrastructure to 
ensure that emerging trends in business and IT 
don't undermine availability and manageability 
needs or negatively impact total costs of owner- 
ship for mission-critical systems. 


Trends in Business Intelligence 

For example, a perfect illustration of how busi- 
ness trends impact IT considerations is the 
continued trend toward increased adoption 
and use of Business Intelligence (BI) solutions. 
As Microsoft continues to help democratize the 
adoption of BI initiatives, and as more and more 
businesses implement BI solutions, increased 
adoption causes a “snowball effect” where more 
and more businesses implement BI solutions to 
keep up with their competition. 

This, in turn, pushes additional demand for IT 
services and infrastructure and drives up require- 
ments for additional processing power and stor- 
age capacity. 


Trends toward Higher Availability 
Another industry trend is the move toward a 
more proactive approach to systems manage- 
ment by IT departments that are seeking to be- 
come more agile. By acknowledging that system 
failures are not a question of “if; but of “when,” 
disaster recovery solutions are increasingly being 
bolstered by High Availability (HA) solutions that 
proactively account for potential outages at the 
system, hardware, or site level. 

In turn, this strategic approach to mitigating 
the costs and impact of disasters and outages 
translates into an increased demand for more 
processing and storage capacity—along with 
the need for sophisticated failover management 
solutions and capabilities. 


Architectural Considerations 

In addition to trends within IT and business, 
architectural choices (which are commonly 
driven by trends within IT) can also have broad, 
long-lasting implications for mission-critical SQL 
Server deployments. 

For example, as single systems have increas- 
ingly been replaced by highly consolidated, 
virtual systems in an effort to decrease costs 
and increase agility, continued demand for 
increased scalability and fault-tolerance is now 
starting to drive another shift, or trend, toward 
utilization of systems within public or private 
clouds. But many database workloads exhibit 
performance and security characteristics that 
aren't amenable to either consolidation or 
deployment within the cloud. 

Therefore, trying to address how trends shape 
architectural developments while deciphering 
how architectural choices impact the ability to 
respond to trends can often lead to a quandary 
where IT professionals are left to second-guess 
which trends and architectural approaches 
to heed or ignore. Happily, keeping things 
simple—and focused on cost—can have a ben- 
eficial impact on determining where to direct 
efforts and energy. 


Ensuring Solution Affordability 
Despite the complexity associated with priori- 
tizing architectural considerations and trends, 
there are a few constants within IT. For ex- 
ample, scalability, availability, and performance 
will always be in demand by end users and 
management—just as continual demand for 
decreased costs will always be a requirement 
from management. And while the constancy 
of these two competing needs may seem like 
a problem, purchasing and leveraging tech- 
nologies and solutions that acknowledge and 
address this reality provide some of the best, 
long-term benefits for IT departments. 

Happily, SQL Server shines in this regard 
due to its high degree of flexibility in meeting 
business needs by simultaneously bundling 
solid OLTP performance and scalability with a 
plethora of BI tools and solutions into the same 
license and solution—allowing IT professionals 
to offer businesses more options and capabili- 
ties for less. As affordable and versatile as SQL 
Server is, understanding SQL Server's depen- 
dence upon storage infrastructure is a crucial 
component of increasing ROI and ensuring 
continued IT success. Consequently, coupling 
SQL Server with an efficient and cost-effective 
infrastructure helps organizations realize cur- 
rent and future objectives and requirements. 


Can Backup Checksums Be 
Used as a Substitute for DBCC 


CHECKDB? 


*m having trouble finding a time when I can 
| run DBCC CHECKDB on my databases 

because of the resources it consumes. I’m 
using the CHECKSUM option on all of my 
backups—is this good enough for me to be able to 
stop running DBCC CHECKDB regularly? 


This is a question I’m asked quite frequently and 
the answer is, unfortunately, no. It’s a well-known 
fact that DBCC CHECKDB (or any of the 
derivative DBCC commands) uses a lot of CPU, 
memory, I/O, and tempdb resources—it’s just the 
nature of what DBCC CHECKDB has to do to 
consistency check a database as fast as possible. 

As databases are getting larger, many people are 
finding that they don’t have a maintenance window 
in which they can accommodate the performance 
degradation that can occur while DBCC CHECKDB 
is running, so they're looking for alternatives. 

When you perform a backup using the CHECK- 
SUM option, the backup system will check the 
page checksums that are present on the data file 
pages being backed up. If a page checksum error 
is found, the backup will terminate with an error 
(by default—this can be overridden using the 
CONTINUE_AFTER_ERROR option). This, of 
course, is sufficient to check whether an I/O subsys- 
tem problem caused corruption in the database, but it 
can’t detect corruptions caused by other problems. 

The reason that using backup checksums isn’t 
a substitute for running DBCC CHECKDB is 


that a data file page could have been corrupted in 
memory (by a faulty memory chip, for example) 
and then the corrupt page written out to disk 
with a new page checksum. Although this is a 
rare occurrence, corruptions like this do hap- 
pen. Simply checking that the page checksum is 
still correct isn’t enough to detect this kind of 
corruption—a proper consistency check must still 
be performed. 

The buffer pool itself will try to catch such 
corruptions before they make it to disk. Part 
of the lazy writer background process, which is 
responsible for keeping free space available for 
pages to be read from disk, will check pages in the 
buffer pool that have a page checksum and haven’t 
been changed in memory since the last time they 
were read from or written to disk. If such a page 
is found to have an incorrect page checksum, high 
severity error 832 will be written to the error log. 

Backup checksums should always be used, but 
the best way to offload the consistency-checking 
workload from your production system is to take 
a full database backup, restore it on another SQL 
Server instance, and run the consistency checks 
there. This procedure also lets you check the valid- 
ity of the backup you just performed. For more 
information, see my blog post “CHECKDB From 
Every Angle: Consistency Checking Options for a 
VLDB” at www.sqlskills.com/BLOGS/PAUL/post/ 
CHECKDB-From-Every-Angle-Consistency- 
Checking-Options-for-a-VLDB.aspx. 


Which Replication Databases 


Can I Mirror? 


e recently had some database corrup- 
tion caused by a faulty drive, and Pd 
now like to add protection to some of 


our databases using database mirroring. It won’t be 
a problem to implement database mirroring on most 
of these databases, but some of the databases are 
involved in transactional replication and I’ve heard 
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conflicting information about whether it’s possible 
to mirror replication databases. Can you shed some 
light on this topic? 


It depends on which database you'd like to mirror 
and which version of SQL Server you're using. 
A transactional replication topology has three 
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QUESTIONS 
ANSWERED 


databases: the publication database, the distribu- 
tion database, and the subscription database (with 
possibly multiple subscription databases). Note that 
I’m not considering peer-to-peer replication here— 
that’s a whole other level of complexity that usually 
precludes mixing in database mirroring. 

Tll start with the easy case: the distribution 
database can’t be mirrored. The mirror database in a 
database mirroring partnership resides on a separate 


The publication database is the easiest 
database to mirror because the Log 
Reader Agent job that runs on the 
Distributor and processes the transaction 
log of the publication database is 
specifically designed to work with 


mirroring on SQL Server 2005 and later. 
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Windows server from the principal database, and the 
Windows server has a different server name from the 
principal Windows server. When a mirroring failover 
occurs (automatically or manually), the mirror data- 
base becomes the principal database and is therefore 
hosted on a different server name. The name of the 
transactional replication Distributor can't change or 
replication stops working. 

The publication database is the easiest database 
to mirror because the Log Reader Agent job that 
runs on the Distributor and processes the transac- 
tion log of the publication database is specifically 
designed to work with mirroring on SQL Server 
2005 and later. There’s a PublisherFailoverPartner 
parameter that’s set to the mirror server name that 
lets the Log Reader Agent automatically connect to 
the new principal database after a mirroring failover 
occurs and applies to the Snapshot Agent job. 

The subscription database can be mirrored on 
SQL Server 2005, but after a mirroring failover 
a new subscription has to be created and fully 


reinitialized from a backup or a snapshot, which 
usually isn’t palatable. 

In SQL Server 2008, a new subscription must still 
be created after a mirroring failover of a subscrip- 
tion database but there’s a new initialization method, 
initialize-by-LSN, which makes this configuration 
much more attractive. The initialize-by-LSN method 
lets the subscription database be initialized by using 
unapplied changes from the distribution database, 
instead of having to completely reinitialize the 
database. 

For example, imagine a mirrored subscription 
database called PaulsDB. The distribution database 
contains a set of changes that have been harvested 
from the publication database, up to time T3, the 
most recent time. The changes up to time T2 have 
already been applied to the principal PaulsDB 
database by the Distribution Agent. The mirror 
PaulsDB database might be a little behind the prin- 
cipal PaulsDB database, and it has received only the 
changes up to time T1 using database mirroring. 

If the principal PaulsDB database then becomes 
unavailable and a mirroring failover occurs, the new 
principal PaulsDB database is missing the changes 
from T1 to T2 (i.e., those that had been applied to 
the old principal PaulsDB database, but not yet mir- 
rored) and from T2 to T3 (i.e., those that hadn’t yet 
been applied to the old principal PaulsDB database). 
As long as the distribution database still has all the 
changes back to time T1, the new subscription that’s 
created can use the initialize-by-LSN method and 
specify T1 as that starting point. 

To summarize, this means that only those rep- 
licated changes that hadn’t yet been applied to the 
old mirror database are required after the mirroring 
failover. To permit this, a minimum retention period 
must be set on the distribution database to ensure 
that replicated changes remain in the distribution 
database for enough time to perform the reinitializa- 
tion. For more information, see the white paper I 
wrote for Microsoft that explains in great detail how 
to combine transactional replication and database 
mirroring—SQL Server Replication: Providing High- 
Availability Using Database Mirroring” (download 
-microsoft.com/download/d/9/4/d948f98 1 -926e-40fa- 


a026-Sbfcf076d9b9/ReplicationAndDBM.docx). 


Is It Possible to Run Out of 
Log Sequence Numbers? 


generating log sequence numbers for transac- 


E ’ve been wondering about the algorithm for 
tion log records, and I’m concerned that with 


a high enough workload it might be possible to 
run out of log sequence numbers. What happens 
in that case? 
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There’s no need to worry because for all practi- 
cal purposes it isn’t possible to run out of log 
sequence numbers. As a bit of background, a log 
sequence number is a three-part number used to 
uniquely identify a transaction log record—con- 
structed from the sequence number of the virtual 
log file (VLF) containing the log record, the log 
block number within the virtual log file, and the 
log record number within the log block. 

This isn’t really important, but what is impor- 
tant is that the VLF sequence number is a 64-bit 
number. Whenever a VLF is reused in the transac- 
tion log, the VLF sequence is increased by 1. So 
let’s do a little math. 

Imagine a transaction log with 65,536 
VLFs, each one 0.25MB in size (not a non- 
sensical situation, depending on how your 
transaction log is being managed—for more 
details on this and on transaction log inter- 
nals, see “Importance of proper transac- 
tion log size management” at www.sqlskills 
.com/BLOGS/PAUL/post/Importance-of-proper- 
transaction-log-size-management.aspx). Each 
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time the log is completely used and wraps around 
to the start, the VLF sequence number will 
increase by 65,536, which is 2 to the power of 
16 (2416). 

A 64-bit number can support 264 values. To 
be able to exhaust the 2^64 possible VLF sequence 
numbers, our example transaction log would have 
to wrap 2464 / 216 = 2448 times. That’s a lot of 
log wrapping. But how much transaction log does 
that equate to? 

Our example log is 65,536 x 0.25MB in size, 
which is 16GB. To wrap that log 2^48 times, you’d 
need to generate 2*48 x 16GB of transaction log, 
which equates to 4 billion petabytes (a petabyte 
= 1,024 terabytes) of transaction log—quite an 
undertaking! 

Even being able to write that log out to a solid 
state drive (SSD) capable of a sustained 600MBps, 
it would take 4 billion petabytes / 600 megabytes = 
approximately 240 million years to generate 4 billion 
petabytes of transaction log. As you can clearly see, 
no one’s in any danger of running out of log sequence 
numbers! Sou 
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n that brief moment after the latest version of 

SQL Server has shipped and before the next 

public release cycle begins, SQL Server pros have 
a perfect opportunity to assess what they want out 
of the current release and what they’re looking for in 
the future. As always, SOL Server Magazine is here 
to help. SQL Server Magazine’s Michael Otey, senior 
technical director, and Sheila Molnar, executive editor, 
joined Ted Kummert, senior vice president of Micro- 
soft’s Business Platform Division, for an exclusive 
interview about SQL Server 2008 R2 and the future 
of Microsoft’s data platform. Ted is responsible for 
setting the strategy for SQL Server, cloud services, 
data programmability, and modeling technologies. We 
spoke with him in mid-summer, before the fall product 
announcements. For an update, see the web-exclusive 
sidebar “Ted Kummert Comments on SQL Server 
Product Announcements,” InstantDoc ID 125984. 


SQL Server Magazine: What do you see as Micro- 
soft’s competitive edge in the relational database 
market? 


Kummert: We talk about building SQL Server in 
terms of the Information Platform Vision, which 
encircles how we think about what we build. We talk 
about a comprehensive set of capabilities that forms 
the foundation of the mission-critical platform. We 
talk about each of the communities we serve in terms 
of the IT pros, the developers, and every end user 
who wants business insights. I look at our overall 
approach: We’re delivering the SQL Server platform 
to the desktop, in the data center, in the cloud. We 


have a very intensive and broad scope. We can start 
with a departmental workload and scale mission- 
critical workloads to the highest levels of scale. And 
we bring all of those capabilities together in one 
product. We think this kind of comprehensive and 
complete approach is unique in the market. 

Now TIl talk about our philosophy in terms of 
how we bring it to market. We feel very good about 
our hardware ecosystem and our partners. They're a 
big part of the overall equation being able to bring 
these solutions to market built on industry-standard 
server platforms with a lot of choice for customers. 
And, given how we approach and price our product 
and how we work with our hardware ecosystem, we 
think it’s a great value for our customers in terms of 
solution costs and total cost of ownership (TCO). 
And certainly in today’s economy TCO really does 
matter to our customers. 


SQL Server Magazine: How critical is business intel- 
ligence (BI) to the future of SQL Server? 


Kummert: For our customers, BI continues to be a 
high priority in terms of investment. The current pres- 
sures on our customers to do more with less and show 
more business value are raising the relative priority of 
investments in BI solutions. BI is fundamentally about 
end users who need to get questions answered. Out 
there in the world of data is the answer to the question 
that’s going to help them move forward and help the 
business move more efficiently. BI is about empower- 
ing end users with tools and capabilities. SQL Server 
2008 R2 and Office 2010 managed self-service BI are 
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revolutionary in terms of empowering end users to 
solve problems on their own. IT still gets to do the 
strategic stuff. We think that’s going to be a great thing 
for our customers. 


SQL Server Magazine: Do you see SQL Server 
gaining market share in 2011 against other high-end 
relational database platforms? 


Kummert: Yes we do. It’s gratifying to see the suc- 
cesses our customers are having using our platform 
for those mission-critical business applications. There 
are a couple of good stories. I think of PREMIER 
Bankcard, one of the largest credit card providers 
in the United States. They’ve been on a journey 
where BI has become a mission-critical part of their 
business decision making. And at the heart of that 
infrastructure is a very large data warehouse—a 17TB 
data warehouse built on SQL Server 2008. And that 
upgrade to SQL Server 2008 gives them the confi- 
dence that it’s going to scale to 30TB capacity in the 
future. I think about First American Title—they run 
their entire title and escrow processing on an applica- 
tion built on SQL Server. It’s a multi-terabyte storage 
application. They upgraded to SQL Server 2008, and 
they take advantage of backup and data compression 
and features for managing the environment that make 
the environment more efficient in terms of the data 
collector and the management views. These things 
help manage and scale the solution effectively and 
efficiently. It’s great to see these successes—our view 
is there’s nothing out there that our product can’t 
scale to. 


SOL Server Magazine: Are these the kind of custom- 
ers that are the driving forces behind the Parallel Data 
Warehouse (PDW) Edition? 


Kummert: Yes, this is certainly the PDW Edition. That 
release is imminent. We’ll be releasing it later in this 
calendar year. This is where we'll add scale-out data 
warehousing to the SQL Server product family. If 
you look at SQL Server and data warehousing today, 
we're out there running at the tens of terabytes. But 
a lot of our larger customers have data warehouses 
that are larger than that. The addition of scale-out 
data warehousing gives us a solution that will scale 
to those data warehouses. So we'll take SQL Server 
to the hundreds of terabytes. And we can be a one- 
stop shop for all of our customers’ data warehousing 
needs—from that large Single Version of the Truth 
data warehouse to the surrounding data warehouses 
in the data marts to the BI tools. You’ve now got one 
platform to manage and one company to deal with. 
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SQL Server Magazine: How do you see the data 
market shifting in the next five years? 


Kummert: BI is a high priority: A lot of our custom- 
ers have multiple solutions in their environment. This 
is an issue in terms of enabling collaboration, and 
we're having a lot of conversations around standard- 
izing BI tools. We’ll continue investments in managed 
self-service BI. The second area is the cloud. Cloud 
computing can enable our customers to focus more 
on the needs of their business because they don’t have 
to spend time and effort managing the infrastructure 
part of the equation. If they're buying Azure from 
us, we're solving that part of the problem for them. 
They can think more about their applications, their 
data, and how to solve their business problems. Cloud 
computing really says, “IT becomes delivered more 
and more like a service.” That’s going to be transfor- 
mational for our customers in the future. Microsoft’s 
been doing cloud computing for quite some time. 
We've been a provider of Internet-scale services. If 
you look at what we’ve been doing in MSN, in Live, 
and in search—we have experience that we bring to 
bear in terms of how we build Azure. 


SOL Server Magazine: How do you see the move 
to the cloud affecting the readers of SQL Server 
Magazine—DBAs, developers, and BI professionals? 
Do you see their roles shifting in the future? 


Kummert: We think it’s part of our respon- 
sibility to our community to shift their roles, 
if you will. They can focus on solutions that 
add business value because we’ve taken some 
of the routine of “just making it work” out of their 
hands. We feel a great deal of responsibility for mak- 
ing the community of IT professionals, database pro- 
fessionals, BI practitioners, and developers successful. 
They're out there with our product every day getting 
problems for the business solved. We want to enable 
them to go up the value chain and focus more on the 
tasks and projects that add strategic value to the busi- 
ness. And cloud computing is a trend that is going to 
enable them to move more in that direction. 


SOL Server Magazine: Do you think that cloud 
computing is easier to adopt in a small organization 
or a larger one? 


Kummert: This is something that’s available for all 
businesses. From the small business that really doesn’t 
want to spend time thinking about and managing an 
infrastructure and just wants to get a business prob- 
lem solved, to the very large enterprise, where the type 
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of investments they 
make are strategic to 
how their business pro- 
cess runs. The cloud 
is real, whether it’s a 
small business or the 
largest enterprise. 


SQL Server Magazine: 
What are Microsoft’s 
plans for extending BI 
to the cloud? 


Kummert: We want the 
BI practitioner or the 
database professional 
to understand that they’re going to be able to use their 
knowledge, skills, and assets as they use SQL Azure. 
We're working forward on a principle of consistency 
that applies to our programming model and tools. 
For instance, with SQL Azure, which just became 
commercially available earlier this year, we currently 
have a subset of capabilities in terms of the overall 
programming model. Developers have been able to 
take an existing app and bring it very easily into SQL 
Azure. We’ve gotten a lot of positive reception to 
SQL Server Management Studio being able to work 
against a SQL Azure environment. Azure is another 
tier in our platform. It’s not a new and distinct tier. 
It’s a part of the end-to-end information platform. So 
we've got a set of capabilities now that orient around 
the application scenarios, but we do intend to offer 
broader services in the SQL Server platform as part 
of SQL Azure—that includes reporting and analyt- 
ics. So that’s definitely on our roadmap. We're not 
announcing specifics today. All Pll say to our readers 
is “Stay tuned!” 


SQL Server Magazine: You mentioned SQL Server 
Management Studio (SSMS). Can SSMS in SQL 
Server 2008 R2 manage SQL Azure? 


Kummert: Yes. We’ve enabled that capability in SQL 
Server Management Studio to manage on-premises 
SQL servers as well as to be able to connect and man- 
age the SQL Server Azure database environment. 


SOL Server Magazine: What’s the feedback you’ve 
gotten so far on SQL Server 2008 R2? What’s the 
adoption rate? 


Kummert: We've seen very strong adoption so far. 
In just a couple of months since we’ve released the 
product we have close to 700,000 downloads, which 
is the largest we’ve ever seen for a new release of SQL 
Server. We’ve gotten a lot of positive reception on the 
capabilities that we’ve delivered. A big part of that 


release is what we delivered in managed self-service 
BI. We have customers in production with a lot of 
interesting applications on that infrastructure, and a 
lot of great feedback on the manageability features— 
multi-instance management and application manage- 
ment. We have interesting conversations going on 
around the new workloads—Master Data Services 
and StreamInsight. We're building on the momentum 
and adoption we’ve seen for SQL Server 2008. 


SQL Server Magazine: What advice can you offer 
readers who don’t keep up with the latest SQL Server 
releases? 


Kummert: We’ve worked hard on a couple of things. 
One is to make sure that the upgrade process is easy 
and seamless. The other is about features that are 
transparent to applications. If you upgrade, these 
are things that, without modifying your application, 
you're just going to be able to take advantage of. In 
SQL Server 2008 there’s some emblematic features— 
data compression, transparent data encryption. 
Without changing your application you can get the 
benefits of the encrypted store and the backup and 
compression. You can go through the upgrade and 
then take advantage of manageability capabilities 
to gain their benefit. So we're making that upgrade 
process simple and delivering new value that you can 
get without investing in new applications or changing 
your existing applications. 


SOL Server Magazine: Our DBA readers are looking 
forward to the next SQL Server release as a strong 
relational database release. What are some of the 
areas that Microsoft is intending to focus on? 


Kummert: We continue to anchor what we do in 
SQL Server around the Information Platform Vision 
[empowered IT, pervasive insight, dynamic develop- 
ment, and mission-critical platform—for more infor- 
mation, see www.microsoft.com/sqlserver/2008/en/us/ 
overview.aspx]. So we think about those four pillars, 
and we think first and foremost about SQL Server as 
the platform for mission-critical applications. That’s 
the foundation. We’ve had a lot of investment in SQL 
Server 2008 and SQL Server 2008 R2 along those 
lines, and I would expect to see us continue to invest 
there. The second area is about the features for our 
IT pros and database professionals: SQL Server’s 
hallmark has been providing the platform with the 
lowest TCO and that’s a continuing commitment. We 
want to enable developers to build richer solutions, 
and we're going to continue to invest along those lines. 
And the last part we talk about is delivering what 
we call “pervasive insight.” It’s about empowering 
every end user with the tools to get their questions 
answered. We’ve invested a lot in the last release on 
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managed self-service BI, which has to do with the 
people who aren't using the formal BI solutions today. 
We don’t want anybody to be confused. We're quite 
committed to enhancing our capabilities in the plat- 
form for those formal, corporate BI solutions. And 
I would expect to see us continue to invest in those 
areas as well. We're pretty excited about what we’ve 
got coming. Stay tuned! We have a lot of exciting 
things to talk about in the future. 


SOL Server Magazine: SQL Server PowerPivot for 
Excel was aimed at the business power user of Excel. 
Our SQL Server authors’ interest in using this tool 
themselves has picked up over the past few months. 
Does Microsoft want to make PowerPivot something 
that SQL Server pros will use? 


Kummert: The managed self-service BI features were 
designed for all end users. And “all end users” doesn’t 
just include business analysts or information workers. 
This includes IT professionals and developers. All of 
them can benefit from the capabilities we’ve delivered 
in managed self-service BI. There are a lot of great 
stories that start with, “Hey we used to have this ad 
hoc compliance process that happened every month in 
a manual way. We get three people in a room to bring 
some data together and put the solution together in 
Excel to answer some question.” PowerPivot enables 
them to build a formal solution, and now it’s just 
there. They’ve deployed it in the environment, the 
data is refreshed automatically, and now they don’t 
have to invest in that manual ad hoc process every 
month. I'd say to a lot of our professionals, “Hey, 
you've got some things like that. Look at those places 
where you're using Excel to analyze data. Think about 
those places as opportunities to put PowerPivot into 
play to enable you to formalize some things you’ve 
been doing ad hoc.” This is a tool our professionals 
can and should be using today. We bring some of the 
capabilities in terms of PowerPivot into our tools for 
BI professional users, and we continue to bring some 
of the features that right now are supported only in 
our formal BI platform down into PowerPivot. We do 
see unification of these environments in the future. 


SOL Server Magazine: One of the surprising things 
in SQL Server 2008 R2 is that Microsoft still offers 
32-bit versions of SQL Server. It seems to be the only 
Server product left that is 32-bit. Do you think that’s 
going to change for the next release? 


Kummert: SQL Server 2008 R2 does support both 
32-bit and 64-bit versions, and the most important 
thing for us is to provide support for the platforms 
where our customers are. And as we look forward 
to our future plans that’s how we’re going to make 
those decisions. 
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SOL Server Magazine: We've heard a lot of talk from 
Microsoft about the “consumerization of IT,” with 
professionals bringing their technology from home 
into the workplace. Do you see this phrase applying 
to SQL Server professionals? 


Kummert: A great example of the consumerization of 
IT would be search. We’ve seen an evolution in search 
where, at first, people could search and find informa- 
tion on the public Internet. 
Now we're enabling that 
search experience in struc- 
tured data and BI appli- 
cations. And think about 
social media—we brought 
the managed self-service BI 
scenario into SharePoint so 
we can enable sharing and 
collaboration. You want to 
be able to find others, build 
on the work of others, and 
SharePoint’s the place for 
you to do that. That also 
allows us to unlock other features of SharePoint; 
social media features: “l like this”; “I want to see 
things that other users like.” We inherit those types 
of things. 


analytics.” 


SOL Server Magazine: Can you talk about Micro- 
soft’s plans for SQL Server regarding unstructured 
data? 


Kummert: A big part of our investment in our infor- 
mation platform and SQL Server is to enable it to be 
able to process all the types and shapes of data you 
want to be able to use and consume in your business 
processes and applications. In SQL Server 2008 we 
added a new set 
of spatial data 
types. In terms 
of unstructured 
data we added the 
filestream column 
type, and we’ve 
seen a lot of posi- 
tive reception of 
customers now 
being able to use 
that column type 
and be able to deal 
with unstructured 
data within that 
same program- 
ming model. 
We're going to 
continue to invest 
in those areas of 
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the platform going 
forward. 


SQL Server Maga- 
zine: What do you 
see as the future of 
SQL Server and 
virtualization? In 
the past a lot of 
DBAs were hesi- 
tant to virtualize 
SQL Server; do 
you think that’s 
changing? 


Kummert: We're 
having this con- 
versation with our 
customers around 
application virtu- 
alization to their 
database environment. And it comes primarily from 
server consolidation, where there are just so many 
benefits for our customers in terms of reducing foot- 
print that provides so many downstream benefits in 
terms of manageability and costs of the solution in 
the environment. We’ve been working to make that 
a great solution on the Microsoft platform—from 
the hypervisor with Hyper-V to the management 
tools with System Center and Virtual Tool Machine 
Manager, and then SQL Server. 


SOL Server Magazine: Does Microsoft use virtual- 
ized SQL Server for anything? 


Kummert: Yes we do. In Microsoft’s Information 
Technology Group we are applying those technolo- 
gies to virtualize those environments and seeing great 
benefits from that. 


SQL Server Magazine: We always like to know how 
Microsoft uses its technologies. 


Kummert: We get a lot of benefit out of Microsoft 
IT as well as our partnership with the other Internet 
properties inside the company. There are thousands 
of SQL Server instances within the Live infrastruc- 
ture. We gain a great deal of learning from that experi- 
ence and we partner very deeply with Microsoft IT as 
we head to the release of a product. We don’t release 
the product until we get applications in production in 
our own environment first. That’s a very important 
part of what we do to get the product ready for our 
customers before we release it. 


SQL Server Magazine: Are there any ways in which 
SQL Server is being used that you didn’t expect? 


Kummert: With SQL Azure we’ve seen some very 
cool stuff. I look at TicketDirect—they’re an event 
management company in Australia, and they provide 
ticketing services. They had an existing application 
running on premises, and they extended it to a Soft- 
ware Plus Services model. And now they’ve added 
the capabilities to use SQL and Windows Azure. And 
what do they get? They get the ability to flex. Say 
there’s a really popular event coming—now they can 
flex into the additional capacity that they need for 
that event. They enabled a new scenario and were able 
to get some of the benefits of the cloud model and 
this “pay as you go” thing almost immediately. I think 
of Esri, a pioneer of geographic information systems 
(GIS). They built a solution called MapIt that’s 
about taking some of the capabilities, exploring the 
geospatial information, and making it more broadly 
available. They brought MaplIt to this platform pretty 
easily. We just introduced spatial data types in SQL 
Azure, and it’s exciting to see how easily Esri was able 
to bring that solution over to Azure. I feel like I’ve got 
this privileged seat where I get to see all these amazing 
things that our customers are able to do. It’s really 
what it’s all about for us. 


SQL Server Magazine: The geospatial data types 
really enable a whole new kind of application. Our 
readers are pretty excited about it. 


Kummert: The ability to analyze data within a spatial 
domain just unlocks insight. It can be as simple as 
being able to download visualized data within a map. 
We just released a mapping control as part of SQL 
Server 2008 R2 Reporting Services. Think of how 
much more compelling it is to be able to look at data 
within that spatial domain rather than in a list of 
addresses. 


SQL Server Magazine: Is there anything you want 
to say directly to our readers—they are a dedicated 
and loyal lot! 


Kummert: I just want to reiterate: The community is 
critical to what we do. What they build around our 
platform—that’s a critical part of how we're success- 
ful. If we don’t make the community successful, we’re 
not going to be successful. We’re not confused about 
this. There are always changes coming. There are 
things we invest in and build; there are new capabili- 
ties; there’s new things coming like cloud computing. 
There’s so much opportunity to become more strate- 
gic in what you do and how you do it. Our commit- 
ment is to this community and to make sure they’re 
successful now and to the future. And we thank them 
for the partnership we’ve had over the years, and we 
look forward to many years to come. SQL 
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Flattening 
Hierarchies 


Convert hierarchies that are represented as 


adjacency lists 


rom time to time I get a request to convert 

a hierarchy represented as an adjacency 

list (parent-child IDs) to a flattened, or 
pivoted, representation. The latter representation 
means that you want the result set to have a row 
for each node in the hierarchy, with a column for 
each of the ancestors of that node, holding the 
ancestor ID. 

As an example, the code in Listing 1 creates 
a table called Employees and populates it with 
sample data. The Employees table holds an adja- 
cency list representation of an employee hierarchy. 
The task that’s the focus of this article is to cre- 
ate a flattened representation of the hierarchy, as 
Table 1 shows. 

I cover two main strategies to address this 
task—one based on joins and another based on 
recursive queries. As usual, I urge you to first try 
to come up with your own solutions before look- 
ing at mine. 


Solution Using a Static Query 
Based on Joins 

The strategy based on joins works reasonably well 
when the degree of the tree (the maximum depth, 
or number of levels) is fairly small and known 
ahead. For example, suppose you know that the 
employee hierarchy won't exceed five levels of 
management in the foreseeable future. You can 
thus address the task at hand with a static query 
that uses four joins. Listing 2 contains a possible 
solution based on the joins strategy. 

Most of the logic in the query defining the CTE 
called C is fairly straightforward—it’s a query with 
four self-outer-joins, where each instance of the 
Employees table represents a different ancestor 
of the initial node. The instance representing the 
initial node is A, the parent of the initial node 
B, and so on. The one thing that might be less 
straightforward is the calculation of the column 
lvl. This column represents the level of the cur- 
rent node, with the integer | representing the root 
node (the CEO in our case), 2 for children of the 
root (direct subordinates of the CEO), and so on. 
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To calculate the current node’s level, you need to 
count how many nodes in the ancestors path lead- 
ing to the current node aren’t NULL. For example, 
for the employee Jiru (employee ID 5), you'll find 
three non-NULL ancestors in its path—A.empid 
= 5, B.empid = 2 (Jiru’s manager), and C.empid = 
1 (Jiru’s manager’s manager, the CEO). The IDs of 
the fourth and fifth ancestors of Jiru (D.empid and 
E.empid) are NULLs. 

To count how many non-NULL ancestor IDs 
exist in each node’s path, I used a new feature 
in SQL Server 2008—table value constructor— 
like so: 
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C SELECT COUNT (Cempid) 
FROM CVALUES(A.empid) , (B 


-empid) , ORE on the WEB 
(C.empid) , (D.empid), (E Download the listings at 
.empid)) AS InstantDoc ID 125930. 


Dcempid) ) AS lvl 


LISTING I: DDL and Sample Data for Employees 


SET NOCOUNT ON; 
USE tempdb; 


IF OBJECT_IDC'dbo.Employees', 'U') IS NOT NULL DROP TABLE dbo.Employees; 


CREATE TABLE dbo.Employees 
C 
empid INT NOT NULL PRIMARY KEY, 
mgrid INT NULL REFERENCES dbo.Employees, 
empname VARCHAR(25) NOT NULL, 
salary MONEY NOT NULL, 
CHECK (empid <> mgrid), 
CHECK (empid > Ø) 
DE 


CREATE UNIQUE INDEX idx_unc_mgrid_empid ON dbo.Employees(mgrid, empid); 


INSERT INTO dbo.Employees(empid, mgrid, empname, salary) VALUES 


(1, NULL, 'David' , $1ØØØð.Øø), 
C "Eitan' $7200.90) , 
@, al "Ina' , $7508.98), 
(Ce 325 "Seraph' , $5000.90), 
G 2 'Jiru' =, $5588.88), 
(Gy 2s "Steve' , $4500.90), 
(Gy sie 'Aaron' , $5888.90), 
(@, 5, "Lilach' , $3500.90), 
@©, 7s 'Rita' , $3000.90), 
Ga, 5, 'Sean' , $3000.90), 
Gils w 'Gabriel', $3000.90), 
a29, 'Emilia' , $2000.99), 
Gs, "Michael', $2000.90), 
(14, 9, "Didi' , $1500.98); 
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This feature lets you use the VALUES clause to 
construct a derived table made out of rows con- 
structed from your own expressions. My derived 
table (named D) consists of five rows, each with 
one column (named empid) holding one of the 
ancestor IDs of the current node’s ancestors. The 
query against the derived table applies a COUNT 
aggregate to count the non-NULL empid values. 
For us, this count represents the current node’s 
level in the tree. 

If you’re working with a version of SQL Server 
prior to 2008, you can use the following expres- 
sion as an alternative: 


C CASE WHEN A.empid IS NOT NULL THEN 1 


ELSE @ END 

+ CASE WHEN B.empid IS NOT NULL THEN 
1 ELSE @ END 

+ CASE WHEN C.empid IS NOT NULL THEN 
1 ELSE @ END 

+ CASE WHEN D.empid IS NOT NULL THEN 
1 ELSE @ END 

+ CASE WHEN E.empid IS NOT NULL THEN 
1 ELSE Ø END ) AS 1v1 


As for the outer query in Listing 2, it has a series 
of CASE expressions responsible for returning the 
ancestor ID in each of the levels of the path lead- 
ing to the current node. Recall that each instance 
in the join represents a certain distance from the 
starting node—A represents the starting node, B 
represents the parent, C the grandparent, and so 
on. But the desired result is supposed to return in 
each of the columns the ancestor with a certain 
distance from the root—not the target node. So, 
for example, the target column levell is supposed 
to hold the ID of the current node’s ancestor in 
the first level of the path. For Didi, who’s in level 
5, the ancestor in level 1 in the path appears in the 
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attribute E.empid. But for Gabriel, who’s in 
level 4, the ancestor in level 1 appears in the 
attribute D.empid. So the CASE expressions 
have the logic and the math to figure out for 
each level in the path which of the attributes 
needs to be returned based on the current 
node’s level. This technique works reason- 
ably well when the degree of the tree is fairly 
small, known, and not planned to increase in 
the foreseeable future. 


Static Solution Using a 
Recursive Query 

Another approach to flattening hierarchies 
uses recursive queries. I'll first demonstrate 
a solution using a static recursive query, 
assuming the degree of the tree won't exceed 
a certain number—for example, five. Then I'll 
show how you can enhance the solution for 
cases in which the degree of the tree can keep 
changing with an unknown maximum. 


LISTING 2: Static Query Based 
on Joins 


WITH C AS 


SELECT. 
empid, 
empid AS A_empid, 
empid AS B_empid, 
empid AS C_empid, 
empid AS D_empid, 
empid AS E_empid, 
SELECT COUNT (Cempid) 
FROM (VALUES(A.empid) , (B.empid), 
(C.empid) , (D.empid) ,(E.empid)) AS 
Dcempid) ) AS 1v1 
FROM dbo.Employees AS A 
LEFT OUTER JOIN dbo.Employees AS B 
ON A.mgrid = B.empid 

LEFT OUTER JOIN dbo.Employees AS C 
ON B.mgrid = C.empid 

LEFT OUTER JOIN dbo.Employees AS D 
ON C.mgrid = D.empid 

LEFT OUTER JOIN dbo.Employees AS E 
ON D.mgrid = E.empid 


amO 


J 
SELECT empid, lvl, 
CASE 1 
WHEN 1v1 THEN A_empid 
WHEN Ivl - 1 THEN B_empid 
WHEN lvl - 2 THEN C_empid 
WHEN Ivl - 3 THEN D_empid 
WHEN Ivl - 4 THEN E_empid 
END AS level1 
CASE 2 
WHEN 1v1 THEN A_empid 


WHEN Ivl - 1 THEN B_empid 
WHEN Ivl - 2 THEN C_empid 
WHEN Ivl - 3 THEN D_empid 
END AS level2, 
CASE 3 
WHEN 1v1 THEN A_empid 
WHEN Ivl - 1 THEN B_empid 
WHEN Ivl - 2 THEN C_empid 
END AS level3, 
CASE 4 
WHEN 1v1 THEN A_empid 
WHEN Ivl - 1 THEN B_empid 
END AS level4, 
CASE 5 
WHEN 1v1 
END AS level5 
FROM C; 


THEN A_empid 
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The first step in the solution is to write a recur- 
sive CTE with the anchor query returning the row 
for the root node and the recursive query returning 
the children of the nodes from the previous round. 
This way the code will query the nodes one whole 
level at a time, starting with the root. With fairly 
simple logic you can calculate each node’s level (1 
for the root; parent’s level plus 1 for a child). You 
can also construct a path consisting of the IDs of all 
ancestors leading to the current node. The path of 
the root consists of just the root’s ID, converted to a 
fixed-length 10-character string with leading spaces. 
The path for a child simply consists of the parent’s 
path plus the child’s ID, again, converted to a fixed- 
length character string. Listing 3 contains the code 
that implements this step. 

Table 2 shows the output of the step. As an 
example, Didi, who has four levels of ancestors, got 
the lvl (meaning level) value 5, and pth (meaning 
path) values 1, 3, 7, 9, and 14. 

The second step in the solution is to have an 
outer query against the CTE Tree extracting the 
individual ancestor IDs from the concatenated 
paths. The implementation of this step, as part of 
the complete solution, is shown in Listing 4. 

A SUBSTRING function is used to extract the 
string representation of each of the IDs. Then the 
expression <string ID> + 0 is used to force implicit 
conversion of the character ID to the numeric ID. 
Finally, NULLIF(<numeric_ID>, 0) is used to con- 
vert an ID of 0 toa NULL. 


Dynamic Solution Using a 
Recursive Query 

The solution in Listing 4 is based on the assump- 
tion that the degree of the tree is 5. In case a known 
maximum doesn’t exist and the degree of the tree can 
keep changing, you can still rely on similar logic— 
you'll just need to dynamically construct the query 
string, then execute it. 

The dynamic solution uses a helper function 
called GetNums that accepts an integer as input and 
returns a sequence of integers of the requested size 
from 1 and on. Use the code in Listing 5 to create 
the GetNums function. 

Listing 6 contains the complete dynamic solution 
for the task. The code in Callout A calculates the cur- 
rent degree of the tree. As you can see, the code uses 
a simple recursive query similar to the one I described 
earlier to calculate the level of each node. The outer 
query then calculates the maximum level in the tree 
and stores the value in the variable @degree. 

The rest of the code is responsible for construct- 
ing the solution query code, storing it in a variable 
called @sql, then executing it. The first part in the 
construction of the solution query code is straight- 
forward—it’s the definition of the CTE Tree, which 
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TABLE 2: Output of Code in Listing 3 


L 


L 


1 1 
2 i 2 
2 1 3 
3 1 3 7 
4 1 3 7 9 
1 4 1 3 il 11 
2 5 1 3 7 9 12 
3 5 1 3 7 g 13 
4 5 1 3 7 9 14 
3 1 2 4 
3 1 2 5 
3 I 2 6 
4 1 2 5 8 
0 4 1 2 5 10 


ISTING 3: Code to Calculate Level and Ancestors Path 


WITH Tree AS 
C 
SELECT empid, mgrid, 1 AS lvl, 
CAST(STR(empid, 18) AS VARCHAR(9@9)) COLLATE Latin1_General_BIN2 
AS pth 
FROM dbo.Employees 
WHERE mgrid IS NULL 


UNION ALL 


SELECT C.empid, C.mgrid, P.1vl + 1, 
CAST(P.pth + STR(C.empid, 18) AS VARCHAR(9@)) COLLATE 
Latinl_General_BIN2 
FROM Tree AS P 
JOIN dbo.Employees AS C 
ON C.mgrid = P.empid 


) 

SELECT empid, Ivl, pth 

FROM Tree; 

ISTING 4: Static Recursive Query 


WITH Tree AS 
K 


SELECT empid, mgrid, 1 AS IVI; 
CAST(STR(Cempid, 18) AS VARCHAR(9@9)) COLLATE Latin1_General_BIN2 
AS pth 
FROM dbo.Employees 
WHERE mgrid IS NULL 


UNION ALL 


SELECT C.empid, C.mgrid, P.1vl + 1, 
CAST(P.pth + STR(C.empid, 18) AS VARCHAR(9@)) COLLATE 
Latin1_General_BIN2 
FROM Tree AS P 
JOIN dbo.Employees AS C 
ON C.mgrid = P.empid 


) 
SELECT empid, lvl, 


NULLIFC(SUBSTRING(pth, 1, 18) + Ø, Ø) AS levell, 

NULLIF(SUBSTRING(pth, 11, 18) + Ø, Ø) AS level2, 

NULLIFC(SUBSTRING(pth, 21, 18) + Ø, Ø) AS level3, 

NULLIF(SUBSTRING(pth, 31, 18) + Ø, Ø) AS level4, 

NULLIF(SUBSTRING(pth, 41, 18) + Ø, Ø) AS level5 
FROM Tree; 
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is the same as in Listing 4. The part that’s new is the 
code in Callout B, which is responsible for construct- 
ing the series of expressions in the outer query’s 
SELECT list—the ones returning the ancestor IDs 
in the different levels. This is the dynamic part that’s 
based on the current degree of the tree. Observe that 
this code queries the GetNums function to get a row 
for each level. The code constructs the right expres- 
sion for each level number. The code uses the FOR 
XML PATH option to concatenate the values from 
the result rows into one string. Finally, the code fol- 
lowing Callout B finalizes the solution query string 
and executes it. 


LISTING 5: Helper Function GetNums 


IF OBJECT_IDC'dbo.GetNums') IS NOT NULL DROP FUNCTION dbo.GetNums; 


GO 


CREATE FUNCTION dbo.GetNums(@n AS BIGINT) RETURNS TABLE 


AS 

RETURN 
WITH 
L@ AS(SELECT 1 AS c 
L1 AS(SELECT 1 AS c 
EZ ASCSELEGT SIRNAS TE 
[ES PAS (SELECT ITAS TG 
L4 AS(SELECT 1 AS c 
L5  AS(SELECT 1 AS c 


UNION ALL SELECT 1), 

FROM LØ AS A CROSS JOIN LØ AS B), 
FROM L1 AS A CROSS JOIN L1 AS B), 
FROM L2 AS A CROSS JOIN L2 AS B), 
FROM L3 AS A CROSS JOIN L3 AS B), 
FROM L4 AS A CROSS JOIN L4 AS B), 


Nums ASCSELECT ROW_NUMBER() OVER(ORDER BY (SELECT NULL)) AS n FROM L5) 
SELECT TOP (@n) n FROM Nums ORDER BY n; 


GO 
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Strategies Summarized 
In this article I demonstrated two main strate- 
gies to flatten hierarchies. I discussed a strategy 
based on joins, which is mainly suitable for 
trees with a small and nonincreasing number 
of levels. I also discussed a more dynamic 
strategy that uses a recursive query to handle 
trees with a changing number of levels and an 
unknown maximum. I demonstrated this tech- 
nique first with a static query, then I showed 
how to apply similar logic to construct the 
query string dynamically after calculating the 
current degree of the tree. These techniques for 
flattening hierarchies are handy for converting 
hierarchies that are represented as adjacency 
lists to a flattened form for purposes such as 
presentation. SQL 
InstantDoc ID 125930 


LISTING 6: Dynamic Recursive 
Query 
(ASPECLARE @degree AS INT; 
WITH Tree AS 
C 


SELECT empid, mgrid, 1 AS lvl 
FROM dbo.Employees 
WHERE mgrid IS NULL 


UNION ALL 


SELECT C.empid, C.mgrid, P.lvl + 1 
FROM Tree AS P 
JOIN dbo.Employees AS C 
ON C.mgrid = P.empid 


) 
SELECT @degree = MAX(1v1) 
FROM Tree; 


DECLARE @sql AS NVARCHAR(1929) ; 
SET @sql = N'WITH Tree AS 
C 


SELECT empid, mgrid, 1 AS lvl, 
CAST(STR(empid, 18) AS VARCHAR (9ØØ)) 
COLLATE Latinl_General_BIN2 AS pth 
FROM dbo.Employees 
WHERE mgrid IS NULL 


UNION ALL 


SELECT C.empid, C.mgrid, P.lvl + 1, 
CAST(P.pth + STR(C.empid, 18) AS 
VARCHAR (989) ) COLLATE Latin1_ 
General_BIN2 
FROM Tree AS P 
JOIN dbo.Employees AS C 
ON C.mgrid = P.empid 


) 
SELECT empid, 1v1 


+ (SELECT 
N'  ,NULLIFCSUBSTRING(pth, ' 


+ CAST(C (n - 1) * 18 +1AS 
NVARCHAR(18) ) 
+ N', 10) + Ø, Ø) AS level' 
+ CASTC n AS NVARCHAR(19) ) 
+ NCHAR(13) + NCHAR(19) 
FROM dbo.GetNums (@degree) 
ORDER BY n 
FOR XML PATH(''), TYPE).value('.[1]', 
"NVARCHAR (1889) ') 


+ N'FROM Tree;'; 


EXEC sp_executesq! @stmt = @sql; 
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DBA’s 


Our 7D Method™ provides a simple 
approach to a solid foundation 


nowing how to restore database files 

isn’t disaster recovery. Scheduling regular 

backups—although that’s a “must-do”— is 
insufficient for proper disaster-recovery planning in 
SQL Server environments. As a SQL Server admin- 
istrator, you need to know how to develop disaster 
recovery procedures and plans that properly protect 
a range of SQL Server—powered business processes— 
databases that span from “never go dark” to “restore 
when you can.” The good news? You don’t need to 
know how to create an enterprise disaster-recovery 
plan. You need to focus on SQL Server protection 
and learn how to contribute SQL Server disaster- 
recovery methods to your company’s more compre- 
hensive business-continuity/disaster-recovery plan. 
Even if your company has no such plan, you want 
to be able to plan and execute a robust process for 
protecting SQL Server assets. 

Just as performing SQL Server backup and 
recovery is different from restoring folders and files 
from a file server, SQL Server disaster recovery has 
unique aspects that—if not considered in a corpo- 
rate disaster-recovery plan—can affect recovery and 
corporate viability. This two-part article touches on 
topics that SQL Server people should consider when 
contributing to an enterprise disaster-recovery plan, 
as well as practices they can adopt for their own SQL 
Server management techniques. 

For our purposes, we're going to apply our 7D 
Method to SQL Server disaster recovery. Originally 
developed to help SQL Server administrators manage 
the life cycle of their databases, the 7D Method is a 
systematic decide and execute process whose seven 
stages include Discover, Design, Develop, Deploy, 
Day-to-Day, Defend, and Decommission. For this 
SQL Server disaster-recovery series, we'll need only 
the first five Ds. In this article, we'll cover Discovery 
and Design, and in Part 2, we'll show you how to 
Develop, Deploy, and maintain your SQL Server 
disaster-recovery plan Day-to-Day. 


The End Is Near! 
Disaster recovery can be a scary term. It implies hor- 
rible things happening: visions of tornadoes carrying 
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your data center to Oz, or your rack of database serv- 
ers sinking beneath floodwaters. In truth, such cata- 
strophic events rarely occur. It’s much more likely that 
you'll be brought down by the simplest of disruptive 
events—a power outage. Whether you're facing fire, 
flood, tornado, blizzard, terrorist attack, pandemic, 
or a straightforward power outage, your job is to keep 
critical SQL Server systems operational, available, 
and connected to the network. 

We can apply the groundbreaking insights of 
Kathleen Lucey—a Business Continuity Disaster 
Recovery (BCDR) thought leader—to our SQL 
Server instances. According to Lucey, there are two 
big misconceptions surrounding disaster-recovery 
planning: first, that preparing for the worst kind of 
catastrophe properly prepares you to deal with any 
lesser disruptions, and second, that smaller orga- 
nizations are fine with the same disaster-recovery 
plan development approach that larger enterprises 
perform. 

Figure 1, adapted from a graphic originally pub- 
lished by Lucey, shows the levels of disruption and 
the relative frequency of occurrence—signified by 
the width of each band—that you might expect from 
minor, significant, serious, and catastrophic interrup- 
tions to your core processes. Notice the frequency of 
catastrophic interruptions. Catastrophes—interrup- 
tions lasting longer than 10 days—have historically 
made up a very tiny percentage (1 to 2 percent) of dis- 
ruptions. The biggest sources of disruptions—those 
that cause up to a full day of downtime—are minor. 

When you're building your plan, focusing on a 
worst-case scenario doesn’t adequately prepare you 
for more common minor interruptions. A cascade of 
minor interruptions can be just as devastating as a 
single major event, putting the average SMB on life 
support. This is just as devastating as if the building 
were quarantined by an anthrax scare! 


Focusing on SQL Server 

If you work for a critical private-sector firm such as 
an infrastructure service provider (e.g., a power com- 
pany), your company probably has a disaster-recovery 
plan in place. Your challenge is to dovetail your SQL 
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Minor Interruptions: 
process failure, data 
corruption, hardware failures, 
employee mistakes 
(1 day downtime) 


Figure | 


Sample interruption categories 


24 November 2010 


Server disaster-recovery plan into that of your enter- 
prise. Critical infrastructure companies are required 
to have contingency plans. Such is not the case for 
most small-to-midsized businesses (SMBs); SMBs 
rarely have responses to severe interruptions prepared 
in advance, nor do they have business-continuity 
specialists—or budgets for them. This article is aimed 
squarely at the SQL Server administrators who must 
assume these duties. 

You're probably wondering how you can possibly 
cram yet another project into your already crowded 
to-do list. Sadly, for some of you, a SQL Server 
disaster-recovery plan might not be optional. If you're 
an infrastructure service provider, financial institu- 
tion, government entity, or health-care company—or 
if you provide critical services to these entities—you 
might have to devise a plan. Your company might 
be required by law or by contract to have a disaster- 
recovery plan. 

The upside? Most of you are already performing 
SQL Server disaster-recovery activities with backups, 
standby and failover servers, log shipping, database 
mirroring, and security policies. These are tactical 
SQL Server disaster-recovery activities; now, you need 
to pull them together into a strategic plan for survival 
and return to pre-incident performance levels. If your 
organization already has a plan, you have something 
that you can integrate into and build on. 

SQL Server disaster-recovery planning must be 
customized for each organization. Think about what 
contribution you and your team can make to crisis 
communication and management. What techniques 
can you develop and share for business-unit and 
technology-recovery procedures? What can you do 
to mitigate the impact of supplier and distributor 
failure? What ideas can you come up with regarding 
relocating and restoring critical IT infrastructure? 
What can you add to testing and validating incident 
recovery plans? 


The First D: Discover 

To align stakeholders and confirm consensus about 
why youre doing the project, you should start 
every project with the Discover phase. In a SQL 
Server disaster-recovery planning project, you should 


understand and acknowledge the real objectives: 
First, ensure that your firm survives any level of inter- 
ruption; second, keep auditors and regulators happy; 
third, keep the boss happy; and fourth, keep your job. 
Any or all of these are valid drivers for a SQL Server 
disaster-recovery planning project. 

A SQL Server disaster-recovery plan can’t be 
hatched in a vacuum. The team you assemble will be 
your SQL Server BCDR Committee. Because your 
focus will be preserving the continuity of your SQL 
Server database operations, you should assemble those 
people who understand database systems, applications, 
OSs, networking infrastructure, security, and upstream 
and downstream data sources. Be sure to include rep- 
resentatives from the business end of the organization. 
Their perspectives are invaluable, so think about the 
process owner and the process participants. The SQL 
Server system you're protecting is the representation of 
someone’s business process, so keep that person in the 
loop. You'll need to select a project manager to orches- 
trate activities and resources. The more experience the 
project manager has with disaster-recovery and conti- 
nuity planning, the better the results, but anyone with 
project-management experience will do. 

The general disaster-recovery planning process 
can be broken down into four steps: inventory (e.g., 
business functions, resources, people); identify pri- 
orities; implement and test the SQL Server disaster- 
recovery plan; and integrate into the enterprise 
disaster-recovery plan. We’ll cover inventory and 
identification in this article; in Part 2, we'll investigate 
implementing and integrating. 

Discover Step 1: Inventory. Your first task is 
to inventory your core business functions and pro- 
cesses. You must approach discovery from a business 
perspective even though you're developing a SQL 
Server disaster-recovery plan. During the Discover 
phase, you’re going to use a top-down approach to 
identify the core business processes and the activities 
that support these core functions. You can then work 
your way down the levels of importance, discovering 
and identifying non-core functions and the activities 
that support these operations. Finally, you'll want 
to identify any cross-references and dependencies 
among core functions and between core and non-core 
functions. When you're done, you'll have a very good 
picture of what the company does. 

Your second task is to map your technology 
resources (e.g., databases, applications) to each of 
the functions you just identified. While you're at it, 
why not include any underlying hardware, software, 
and networking infrastructure, plus anything else you 
might know about, such as special instructions for 
activation and use? In a subsequent step, you'll have 
these business functions prioritized for you by the 
committee, and when that happens, you'll know what 
technologies support each business function. 
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Your third task is to inventory your personnel and 
their job roles. Who does what? Have them tell you, 
and don’t be surprised if you find out that Joe isn’t 
just a programmer—in fact, he’s been doing database 
backups for over a year now. A finding like that 
would be great because redundancy in SQL Server 
skills is necessary as part of a continuity plan. If 
you don’t already have it, skills cross-training needs 
to be part of the implementation plan. But for now, 
simply find out who does what, who can fill in for 
whom, and who has overlapping skills and which 
skills have recurring certification requirements. You 
get extra points if you already have someone (besides 
yourself) with adequate skills who can step in and 
keep the place operational. (This is called the hit by 
a bus plan.) 

Your fourth task is to inventory continuity fea- 
tures and functions already in place. Remember, back- 
ups don’t equal continuity! If you’re backing up but 
not testing the restore, shame on you! Restorability is 
what matters. Do you have standby and/or failover 
servers? Are you doing log shipping or database mir- 
roring? Windows clustering counts, but when that 
cluster fails, it’s going to take much longer to get the 
cluster (and the databases it houses) operational again 
than to get a single server running. Are your data- 
bases virtualized? If a virtualized database or virtual 
machine (VM) fails, how quickly can you start a new 
VM with SQL Server intact and running, and how 
much data loss will it entail? Are your SQL Server 
machines hosted at a hardened facility outside your 
geographic area, or are they on-premises? 

Your fifth task is to identify hardware and net- 
working infrastructure that needs to be upgraded or 
replaced, and prioritize the list. Identify physical plant 
components (generators and security systems, espe- 
cially) that would be essential for local and/or remote 
continuity, and prioritize them. Identify key personnel 
and skills training still needed, and prioritize that list. 

Finally, no inventory would be complete without 
a full materials inventory. You might already have 
something like this in your files. It’s normal to have to 
do a materials inventory—hardware, client comput- 
ers, licensed and non-licensed software, and network- 
ing components—on an annual basis. Only this time, 
include everything on the physical premises that’s 
essential for continuity: air conditioning, security 
access, work space accommodations, and so on. Make 
sure you can locate copies of all software, make sure 
you have backup copies of all software installation 
disks and registration keys, and make sure you have a 
record of the details of each hardware and network- 
ing component so that when they fail you can quickly 
replace them with similar or compatible models. Your 
Property and Casualty insurance policy might have 
an asset inventory that can serve as a starting place. 
Consider downloading free inventory-discovery tools 
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for identifying workloads and devices for server 
consolidation. 

Discover Step 2: Identify Priorities. The most 
important databases might not be the largest or those 
with the highest number of transactions executed 
against them in a day’s time. Rather, they’re the ones 
that support the most critical business functions. 
During the first part of the Discover phase, you 
inventoried the core business functions. Using this 
list and your mapping of resources that support these 
core functions as a guide, you need to determine the 
categories of data-availability criticality: 

e Class A—What data must be available 
247X365? 
e Class B—What data must be available within 

1 day? 

e Class C—What data must be available within 
3 days? 

e Class D—What data can wait a week, or up 
to a month or more, before it can be brought 
back online? 


What do you need to complete the plan, to fill in 
the gaps in each list? Does your office have genera- 
tors that can kick in during power failures (the most 
common type of continuity interruption), and how 
long will they keep the systems alive and functioning? 
What about air conditioning? How long will the AC 
stay operational following a power outage? 

What about weather priorities? If your company 
is located in a part of the country that’s subject to 
weather events (e.g., blizzards, hurricanes, tornadoes), 
what’s your “foul-weather plan”? It’s not reasonable 
to ask your staff to navigate icy or flooded roads, 
risking life and limb, to get to the office; neither is it 
reasonable to close the office when you don’t have to. 
A simple plan that includes switching to teleworking, 
using web-enabled apps for accessing data stored on 
servers located at hardened facilities—with the staff 
remotely accessing that data from home when the 
roads are impassible—solves both situations. Properly 
documented work processes allow for absenteeism 
with minimal service interruption, but it takes plan- 
ning. The great news is that a plan like this is adaptable 
for use in a disaster situation. If the building/block/ 
campus becomes unusable, the data and servers are 
safely tucked away at the hosting facility, your staff 
is working from home or from some satellite office 
that you can quickly set up, and business continues— 
probably more slowly than usual but minimizing the 
impact to your company’s revenue cycle. 

Before you can go to the Design stage or imple- 
ment any of your findings, you must have a visible 
buy-in commitment from senior management. In a 
large organization that has a conventional plan in 
place, the decision makers can sleep better at night 
knowing that they’re ready for a disaster that might 
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never occur—and employees can feel better knowing 
that the company is looking out for them. That’s not 
how it works in an SMB. The cost/benefit ratio of a 
traditional disaster-recovery plan is seldom feasible 
for them. Business owners and managers are driven 
primarily by their bottom line and would rather risk 
interruption than pay for talent and technology to 
properly protect information assets. It’s a challenge to 
answer the question, “How will a SQL Server disaster- 
recovery plan help my business?”—particularly when 
an SMB owner asks the question. Even an SMB that 
requires a high level of trust from its customers might 
be reluctant to put a disaster-recovery plan in place 
unless it’s mandated by a key customer as a contract 
requirement. The SMB incurs cost for something that 
few, if any, customers will ask about. But, if an SMB 
is to survive disruptions, it needs a tested plan. And 
that plan, or components of the plan, should be part 
of a technology budget ever year. 


The Second D: Design 

Now that you have an idea of what you need to build a 
good SQL Server disaster-recovery plan and continu- 
ity scheme, it’s time to bring all the lists together into 
a plan and prioritize the components. You're creating 
a SQL Server disaster-recovery plan—not a disaster- 
recovery plan for your entire organization—so the 
order in which you'll execute tasks might not be the 
same as you would expect to find in a full enterprise 
disaster-recovery plan. 

Web Table 1 (www.sqlmag.com, InstantDoc ID 
126089) shows a sample template you can use during 
both the Discover and the Design phases. In the first 
section of the template, you can list your organiza- 
tional business processes and functions by name and 
classify them as core or non-core. Create additional 
classes such as core support if that helps you get your 
arms around the big picture. In the second section, 
you can list the technology resources that you have in 
your shop. Then, cross-reference business processes to 
technology resources, using either resource names or 
associated numbers. Fill out the rest of the Inventory 
Resources section, using information you've collected 
about personnel/job roles/skills, continuity features 
already in place, infrastructure that needs upgrading 
or replacing, and the materials inventory. Nothing 
is sacred about this template; you can modify it to 
your own specific needs, then use it as a checklist 
once you have all your Discover phase inventory lists 
recorded. 

Next, prioritize the data based on the class of 
required availability, ranging from Class A to Class 
D. For each data set, include the name of the data 
repository in as much detail as you need, even to the 
level of server_name.database_name.schema_name 
.tablename. If the data resides outside the SQL 
Server system, include the full path and file name, or 


at least the file folder name. List whatever continu- 
ity features you already have in place for each data 
set, and describe any upgrades or enhancements you 
feel are necessary to ensure the appropriate level of 
continuity. 

Finally, perform a cursory design of your protec- 
tion scenario, once again built on the data availability 
A-D classification. For all classes of data, list the 
business processes supported, names of the technol- 
ogy resources involved (not just SQL Server systems 
but network infrastructure, file servers, VPNs, web 
servers, and so on), what continuity enhancements 
will be needed to meet the availability classifica- 
tion, and the funding requirements and timelines 
involved. 

Armed with a completed spreadsheet that 
describes the current situation and what’s needed 
to meet data availability requirements, you're now 
prepared to make a strong case for funding your 
SQL Server disaster-recovery plan. First, take your 
report to the SQL Server BCDR Committee and 
discuss with them and the project manager how 
your SQL Server plan dovetails into the enterprise 
disaster-recovery plan. Then, prepare to present the 
business case for proper SQL Server protection to an 
executive sponsor (or sponsors) whose budgets will 
fund the plan. 


Next Time, We’ll Go 3D 

Here’s the reality of the situation: No matter how 
good your SQL Server disaster-recovery plan is, if 
the rest of the company falls apart, your SQL Server 
disaster-recovery plan might never be activated! 
Note that, in most companies, a disaster must be 
declared before people actually turn to their “enter- 
prise” disaster plan. That’s why you classify your 
critical data and develop your incident-response 
processes. 

Your company might never declare and activate 
its larger plan, but your plan still has upstream and 
downstream dependencies. For example, suppose 
you can connect to the power grid, but the power 
company is down. Or you can connect to the Inter- 
net, but the DNS servers are offline. The Internet 
is crippled, and your work is minimized. Does this 
mean you shouldn’t prepare? Absolutely not! 

As you'll see in Part 2, a simple approach 
will give you a solid foundation for a range of 
responses. By completing the Develop, Deploy, and 
Day-to-Day steps as the last part of this exercise, 
you'll have a viable framework for an interruption 
of any severity level. You'll have a direct line of 
sight into the most important activities in docu- 
menting a SQL Server disaster-recovery plan. Stay 
tuned for “The Smart DBA’s Guide to SQL Server 
Disaster Recovery, Part 2”! SQL 
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Confusing High Availability with 


Disaster 
Preparedness 


Don't fall prey to common misconceptions 
about protecting your data 


hen considering a proactive approach to 
disaster recovery, it’s only natural to look 
to high availability solutions as a way to 


cope with system or data center outages. However, 
although high availability solutions are a fantastic 
complement to any disaster recovery plan, substitut- 
ing high availability solutions for disaster recovery 
can be fatal. 


Highly Available Bad Data 

High availability isn’t a replacement for a fully baked 
disaster recovery plan because high availability solu- 
tions just make data available—even if the data is 
disastrously bad. For example, consider an inventory- 
management solution atop a three-node failover 
cluster. This solution also uses synchronous database 
mirroring to keep remote copies of data synchronized 
in another location to protect against data center 
failure. With system-level redundancy in the form of 
clustering, and data-level redundancy provided by 
mirroring, many organizations mistakenly assume 
that they’ve implemented a solution that protects 
them from disaster. 

But imagine a scenario in which a software glitch 
requires a developer or DBA to manually set the 
inventory level of a certain problematic product 
down to a quantity of zero. Now, suppose that dur- 
ing this dicey operation, the person executing this 
command forgets to add a WHERE clause to his 
or her UPDATE statement—and accidentally sets 
the inventory level for all products to zero. High 
availability solutions can’t provide protection in a 
scenario like that. In fact, they just make the problem 
worse because they'll simply duplicate the bad data 
(or operation) out to multiple sites. The result would 
be that you now have copies of bad data in multiple 
locations. High availability solutions are also prone 
to these same duplication problems when it comes to 
sabotage by disgruntled employees and corruption 
by hackers. 
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Furthermore, if you’ve never had the opportu- 
nity to recover from a situation like this, it’s easy 
to fall prey to the notion that you can just use the 
transaction log file to recover back to the point in 
time before the disaster occurred. However, such a 
simplistic solution rarely works with highly trafficked 
systems. In our example, in which inventory levels 
get set to zero for all products, this problem will 
quickly manifest itself when someone tries to place 
an order—because inventory will be unavailable. 
But assuming that you're able to recover inventory 
levels to their pre-glitch levels, what about inventory 
additions entered into the system by the receiving 
department while you're working on a recovery for 
the “zero-out”? 

High availability systems only make your data 
available; they don’t control whether the data is cor- 
rect. Therefore, if you don’t have a well rehearsed and 
regularly practiced disaster recovery plan (which can 
benefit tremendously from a third-party log reader 
agent) to cope with bad data, you might be left with 
highly available bad, or busted, data. 


Non-Trivial Implementation 
High availability solutions impose an additional layer 
of complexity. This complexity, in turn, requires 
additional management costs and considerations, and 
implies the possibility of increased system require- 
ments and the potential for additional performance 
overhead. More important, with the addition of 
increased complexity comes the heightened risk of 
problems or disasters simply because there are more 
moving parts and dependencies in play. 

In other words, increasing system or data avail- 
ability is non-trivial and imposes risk, although the 
benefits of properly implementing a high availability 
solution far outweigh the risks. For example, failover 
clustering provides tremendous benefits in terms of 
system availability, but it’s an expensive and non- 
trivial solution to implement. 
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Database mirroring is another non-trivial solu- 
tion that provides tremendous benefits from an 
availability standpoint when implemented correctly. 
But to ensure correct implementation, hardware and 
network throughput need to be adequately sized to 
prevent a big SEND queue from degrading perfor- 
mance during peak operating times. 

Consequently, with any high availability solution, 
make sure to size and plan correctly—and consider 
how your disaster recovery plan needs to address any 
problems that might stem from your high availability 
solution, causing performance problems or distracting 
you from your primary focus of keeping data safe. 


Microsoft SQL Azure promises some exciting oppor- 
tunities and potentials for organizations hoping either 
to benefit from decreased management costs or to 
(eventually) be able to scale without massive out-of- 
pocket costs. But although SQL Azure comes with 
a 99.9 percent up-time guarantee and service level 
agreements (SLAs) that ensure the protection of cus- 
tomer data in the case of SQL Azure hardware and 
system failures, there’s still a chink in the armor. 


Because of the non-trivial tasks associated with 
decoupling the physical implementation details of 
how SQL Azure works from the underlying, physical 
storage engine, there’s currently no available access 
for end users to the transaction log file. Therefore, 
end-users can’t issue BACKUP or RESTORE com- 
mands. Microsoft ensures that SQL Azure data is 
protected from system failures, but there’s no way for 
SQL Azure subscribers to protect themselves from 
human errors or software glitches—serving as a per- 
fect example of how a highly available solution can be 
susceptible to disaster recovery limitations. 

omple itar’ olution: 

When properly implemented, high availability solu- 
tions complement your data protection efforts to 
facilitate effective disaster recovery plans. Don’t let 
the considerations outlined here deter you from being 
proactive and implementing a high availability solu- 
tion. Just don’t assume that high availability solutions 
will be problem-free, and don’t mistake high availabil- 
ity solutions as a replacement or substitute for a well 
rehearsed, regularly updated, and fully documented 
disaster recovery plan. SQL 
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SQL Server 
Query Basics 


Common-sense advice for identifying your 
poorly performing queries 


s a consultant, I see many aspects of 
Ae Server hardware configurations and 

application software that can cause daily 
performance problems. But one of the challenges 
that DBAs face the most is determining which state- 
ments executed in the SQL Server instance have the 
most damaging impact on performance. I encounter 
this challenge so often, in fact, that I feel compelled 
to revisit this topic—if only to put your mind at ease 
and tell you that solving this problem is certainly not 
beyond your capabilities and can take much less time 
and energy to solve than you realize. 


All in the Details 
In several previous articles, I’ve detailed techniques 
for capturing and processing this kind of informa- 
tion. There are essentially two methods you can use 
to collect the data and format it in such a way that 
you can make sense of it. The first method is using 
SQL Trace to collect the statements issued to the 
SQL Server instance and parsing the data with some- 
thing like the SQL Signature UDF (as in “Finding 
Your Top 10 SQL Server Queries,” InstantDoc ID 
100121) or using an external application, such as the 
RML utilities (as discussed in “Trace Reporting with 
RML Utilities,” InstantDoc ID 100670). The second 
method is using the built-in DMVs, such as the sys 
.dm_exec_query_stats (as outlined in “Are Your SQL 
Server Statements Performing Well?” InstantDoc ID 
97761 and “Quickly Find Your Worst-Performing 
T-SQL Statements,” InstantDoc ID 100201). 
Because those resources are already available, I 
won't go into detail about how to collect or parse the 
data. Instead, I'll concentrate on what you should 
look for—and why you should look for it—to maxi- 
mize the time and energy you spend on finding the 
statements to tune first. 


Back to Basics 

Regardless of the method you use to obtain the data, 
your goal is to find the statements that are affecting 
your performance the most. You might think that 
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sounds like a difficult task, but it’s much simpler than 
you might imagine. Although the DMV might give 
you a large selection of metrics, most people use a 
combination of just five metrics to start with. These 
metrics are Number of Executions, Reads, Writes, 
CPU, and Duration. 

For the purpose of this process, let’s ignore Writes 
for now. And although Duration can be a valuable 
piece of information, it can also be somewhat mis- 
leading. For example, if a lot of blocking is occurring, 
Duration can vary wildly for any given statement. 
So, if the duration of a particular statement is high, 
it could be due to blocking and might not require 
tuning at all. Duration is also usually a byproduct of 
high reads or CPU usage, and if we reduce those, we 
reduce Duration. So, let’s set that metric aside, too. 

A basic concept you must grasp is that any one 
of these metrics by itself doesn’t reveal enough infor- 
mation to let you see its true impact. To get a clear 
determination of which statements are affecting your 
system the most, you really need a combination of 
executions and one other metric. Think about that for 
a moment: A single query that requires a million reads 
to finish is pretty expensive, right? But if the query 
is executed only once per hour, it might not even be 
close to the top in terms of overall resource utiliza- 
tion. However, another query that takes 20,000 reads 
per execution but runs 1,000 times an hour would 
probably be a more likely candidate for tuning first. 
Remember to always view metrics in combination 
so that you get a better overall perspective of what’s 
affecting the system the most. That way, you get a 
clearer idea of what you need to tune first to make 
the greatest impact. 


Key In on Your Metrics 

Any statement that’s executed inside SQL Server 
will consume physical resources to perform its work. 
These resources are mostly composed of Memory, 
CPU, and I/O. Because these resources are always 
limited and must be shared by other statements being 
executed, you want to ensure that each statement 
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utilizes as little as possible. Generally, the fewer 
resources utilized, the more performant the statement 
will be. The two metrics I like to focus on most are 
Reads and CPU. A read is an 8KB page being read 
from either memory or disk. If you're using SQL 
Trace, only Logical Reads are reported. But if you use 
the DMV, you can choose from Logical or Physical 
Reads, depending on what you're trying to trouble- 
shoot. CPU is the number of milliseconds of actual 
processor time the query consumes during execution. 
Don’t confuse that with Duration, which is the overall 
time to completion and might include idle time. 

For determining which statements utilize server 
resources the most, I prefer to first concentrate on 
Logical Reads. The more reads a statement does, 
the more overall work must be done to complete the 
task—in contrast with a comparable statement that 
has fewer reads. That’s why I prefer to start my list 
by looking at statements that, in a given time period, 
issue the most reads in total when taking into account 
the number of executions times the individual (or 
average) reads per execution. 

In reality, the way in which the pages are pro- 
cessed can play a significant role in how much work 
is necessary to process the pages. For example, if the 


Hit 
INS 


statement uses a hash join, it might require much 
more CPU resources than a comparable statement 
with the same number of reads that uses a nested loop 
or merge join. For that reason, you must also consider 
CPU usage, and you might want to sort on this metric 
in much the same way. Then, take your top queries 
from each list to determine which ones you want to 
have in your final list. 


No Excuses 
By following these guidelines and utilizing the tools 
referenced here, you should be able to easily identify 
which statements you need to tune first to make the 
most positive impact on your system. Don’t lose sight 
of the fact that, in many cases, 80 to 90 percent of 
performance problems are caused by just a handful 
of statements or types of statements. By identifying 
and addressing the worst ones first, you can often 
make a significant impact with just a small amount of 
time and effort. Now you don’t have an excuse when 
your boss asks you to find the poorly performing 
statements in your SQL Server instance. Of course, 
tuning the statements might be a different matter 
altogether! SQL 
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Building . 


Dynamic. , 
Data Dictionaries 


Run these stored procedures 


ecently I was working on a new database 
R for a client, and I was asked to 

prepare a data dictionary of the database. 
Instead of creating a dedicated single-use document 
containing the data dictionary, I thought it might be 
more convenient to design a general mechanism for 
generating data dictionaries in a database. Such logic 
could be encapsulated within stored procedures, and 
the logic would be reusable. 

In terms of output, I thought it would be helpful 
to see the dictionary information in the form of an 
online SQL Server Reporting Services (SSRS) report. 
In this way, a data dictionary could be hosted on a 
company’s departmental reports site, and different 
team members could refer to the data dictionary 
report as needed. In addition, because SSRS allows 
reports to be saved in Microsoft Excel, Microsoft 
Word, XML, and other formats, the data dictionary 
information could easily be exported into another 
report or document. 

In this article I present a set of stored procedures 
that you can use to generate a data dictionary of 
schemas and tables within a database. I also introduce 
a simple SSRS report that you can use to publish 
the data dictionary online. I tested the accompany- 
ing code in SQL Server 2008; it also works in SQL 
Server 2005. 


Background 
All data dictionary information within SQL Server 
is set and accessed through extended properties. 
According to SQL Server Books Online (BOL), 
extended properties let you “add text, such as descrip- 
tive or instructional content, add input masks, and 
add formatting rules as properties of objects in a 
database or of the database itself. For example, you 
can add an extended property to a schema, a schema’s 
view, or to a column in the view. Because extended 
properties are stored in the database, all applications 
reading the properties can evaluate the object in the 
same way. This helps enforce consistency in the way 
data is treated by all the programs in the system.” 
Extended properties have several different levels. 
SQL Server BOL explains these levels as follows: 
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“For specifying extended properties, the objects in a 
SQL Server database are classified into three levels: 0, 
1, and 2. Level 0 is the highest level and is defined as 
objects that are contained at the database scope. Level 
1 objects are contained in a schema or user scope, 
and level 2 objects are contained by level 1 objects. 
Extended properties can be defined for objects at any 
of these levels.” 

To set an extended property, you use a T-SQL 
command such as sp_addextendedproperty (to add 
a new property) or sp_updateextendedproperty (to 
update an existing property). Similarly, you use the 
sp_dropextendedproperty command to drop an exist- 
ing extended property in the database. 


The spAddToDataDictionary 
Stored Procedure 

I created the spAddToDataDictionary stored proce- 
dure as a simple wrapper around some logic 
to add an extended property in the database 
(or update it, in cases in which the property 
already exists). Web Listing 1 (www.sqlmag 
.com, InstantDoc ID 125905) contains the 
code for the spAddToDataDictionary stored 
procedure. You can use this procedure to add new 
data dictionary entries for tables and table columns 
in a database. It’s a quick way to load data diction- 
ary information into a database if it isn’t already 
available. 

The spAddToDataDictionary stored procedure 
accepts arguments for the schema name, table name, 
and column name, as well as a description. It auto- 
matically handles the level 1 (e.g., schema) and level 
2 (e.g., table) specifications as appropriate. As an 
example, suppose your database contains a table 
definition for vehicles defined as follows: 


CREATE TABLE dbo.Vehicle 


C 
VehicleID UNIQUEIDENTIFIER NOT NULL 
PRIMARY KEY, 
VehicleName VARCHAR(32) NOT NULL 
J 
ee) 
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You might use the following set of stored procedure 
calls to enter the corresponding data dictionary 
entries into the database: 


-- Set the Dictionary Entry for the 
Table 

EXEC [dbo].spAddToDataDictionary 'dbo', 
"Vehicle', 'A Table Listing the 
Different Vehicles'; 


-- Set the Dictionary Entry for the 
Table Columns 

EXEC [dbo].spAddToDataDictionary 'dbo', 
"Vehicle', 'VehicleID', 'A Unique ID 
for the Vehicle'; 


EXEC [dbo].spAddToDataDictionary 'dbo', 
"Vehicle', 'VehicleName', 'The Vehicle 
Name'; 


Notice that the schema name is dbo and the table 
name is Vehicle. For the first call, you’re specifying 
data at the table level, so it isn’t necessary to specify 
a column name. In the second two calls, you're speci- 
fying column-level information, so a column name 
(VehicleID and VehicleName, respectively) must be 
passed in. 


The spGenerateDataDictionary 
Stored Procedure 
After you enter the data dictionary table and col- 
umn information into the database, you can use the 
spGenerateDataDictionary stored procedure to gen- 
erate data dictionary report output. Web Listing 2 
contains the code for this procedure. 

The spGenerateDataDictionary stored proce- 
dure retrieves information from several catalog views 
(e.g., sys.schemas, sys.tables, sys.columns, sys.types) 
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and pulls in information about schemas, tables, 
columns, and column types to the database. The 
procedure then uses the sys.extended_properties 
catalog view to obtain the extended property 
descriptions for these objects. (These extended 
property descriptions provide the information for 
the data dictionary table descriptions and table 
column descriptions.) 

The procedure has two optional parameters: 
@SchemaName and TableName. If the param- 
eters aren't specified, they both default to wildcard 
({All]), meaning that all schemas and/or all tables 
are included. 

As an example, suppose you created the sp- 
GenerateDataDictionary stored procedure within the 
SQL Server 2008 AdventureWorks database. Running 
the following command 


EXEC [dbo]. [spGenerateDataDictionary] ; 


would return information about all the schemas and 
tables in the AdventureWorks database. 
Similarly, if you ran the command 


EXEC [dbo]. [spGenerateDataDictionary] 
"HumanResources'; 


the procedure would return only information 
about tables in the HumanResources schema of the 
AdventureWorks database. 

Similarly, the following command 


EXEC [dbo]. [spGenerateDataDictionary] 
"HumanResources', 'Department'; 


would return only information about a particular 
table (i.e., Department) within the HumanResources 
schema of the AdventureWorks database. 

Figure 1 shows the output of the spGenerate- 
DataDictionary stored procedure. Note that the 
value in the first column is simply the server name 
(@@SERVERNAME). For each table group in the 
output, a separate row describes the information for 
that table (corresponding to ColumnName = [N/A]), 
with one or more rows for each table field. Separate 
columns contain the schema name, table name, data 
type, length, nullability, and description—as you 
might expect. The XType column identifies whether a 
particular column is a primary key (PK), foreign key 
(FK), or both (PK, FK). 

If a data dictionary description isn’t available 
for a particular table or table column (e.g., if the 
extended property for that table or column was never 
set), the stored procedure sets the Description field 
in the output to [Missing Definition]. Indeed, you 
can run the spGenerateDataDictionary procedure 
routinely (e.g., once a week) and look for occurrences 
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of [Missing Definition] in the output to find objects 
that might not be fully notated in the data diction- 
ary. You can then use the spAddToDataDictionary 
stored procedure or another mechanism to manu- 
ally enter the missing information for these entities. 
This approach is a good way to keep the diction- 
ary up-to-date. When a column has a type such as 
XML, VARCHAR(MAX), NVARCHAR(MAX), or 
VARBINARY(MAX), the Length (MaxLength) field 
will show a value of -1, because SQL Server might not 
know the actual column length in these cases. 


Building the Data Dictionary 
Report 

I created an SSRS report that you can use to publish 
the data dictionary. You can use SQL Server 2008 
Business Intelligence Development Studio (BIDS) to 
open the SSRS report project, called DataDictionary- 
ReportProject.slIn, which contains the report called 
DataDictionaryReport.rdl. This simple report form 
lets you call the spGenerateDataDictionary stored 
procedure with appropriate parameters and display 
the output onscreen. 

Within the RDL file (i.e., DataDictionaryReport 
.rdl), you need to edit the data source definition to 
point to the server and database you want to use, as 
callout A in Listing 1 shows. Note that because the 
report form uses integrated security, you don’t need to 
specify a user ID and password. If the data dictionary 
report will be published to a remote reports site on a 
different server in your enterprise, you might need to 
tweak the connection information and/or security cre- 
dentials to enable connection. If you aren’t using an 
administrative account, you'll need to grant explicit 
database access rights to the account you use to con- 
nect to the database. The account should have execute 
rights to the spGenerateDataDictionary procedure 
within the database, as well as to any other procedures 
your reports call. 

When you run the report, you'll be prompted to 
select a schema, as Figure 2 shows. As you can see in 
Figure 2, the schemas for the AdventureWorks data- 
base include dbo, HumanResources, Person, Produc- 
tion, Purchasing, and Sales. When you select a specific 
database schema, the report output is filtered to show 
only the tables within that schema. If you don’t need 
to filter the schema name, you can select [All], which 
includes all schemas in your report’s database. 

After you select the schema, the Table param- 
eter’s drop-down menu will include the available 
table names for the selected schema. As Figure 3 
shows, the tables for the HumanResources schema 
include Department, Employee, EmployeeAddress, 
EmployeeDepartmentHistory, EmployeePayHistory, 
JobCandidate, and Shift. Again, you can select [All] 
to include all the tables within the selected schema. 
Click View Report to generate the report. 
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LISTING |: Editing the Data Source Definition 


<DataSources> 
<DataSource Name="DataSource1"> 
<ConnectionProperties> 
<DataProvider>SQL</DataProvider> 
<ConnectString>Data Source=JOHNDELL\SQL2998; Initial 
Catalog=AdventureWorks</ConnectString> 
<IntegratedSecurity>true</IntegratedSecurity> 
</ConnectionProperties> 
<rd:DataSourceID>2976979c-43a4-45f2-af7b-d5fObc4f8c13 
</rd:DataSourceID> 
<rd:SecurityType>Windows</rd:SecurityType> 
</DataSource> 
</DataSources> 


Bch 


IB DataDictionaryReport.rdl - Report Preview "W ME a 
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Figure 3 


Selecting a table 


Get Started 

Because data dictionaries are useful for both DBAs 
and developers, it’s handy to have a quick mechanism 
for generating them. This article describes some basic 
stored procedures that you can use to build a data dic- 
tionary of tables in a SQL Server 2008 database, then 
use a simple SSRS report to report on that dictionary. 
Although this article is limited to tables, you can easily 
enhance the underlying stored procedures to support 
other objects in the database (e.g., views, user-defined 
functions). You can publish a data dictionary report 
either locally or to a departmental reports site; your 
organization’s DBAs and developers can then access 
the information as necessary. For more information 
about creating data dictionaries, see “Use Extended 
Properties to Create a Data Dictionary,” InstantDoc 
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»LINQ 


Still a powerful tool for working with SQL 


Server data 


couple of years ago I wrote an initial intro- 
Ae to Language-Integrated Query 
(LINQ) to SQL (see “LINQ to Your 
SQL Server Data,’ www.sqlmag.com, InstantDoc 
ID 98205). One of LINQ’s original goals was to let 
application programmers replace the use of T-SQL 
with Visual Studio and LINQ queries that could be 
converted into T-SQL-based queries. LINQ to SQL 
soon fell by the wayside as Microsoft pushed its ADO 
.NET Entity Framework as the up-and-coming data 
access technology. However, LINQ still has a place 
in the application development architecture, and 
Microsoft continues to support the LINQ to SQL 
subset—albeit with few, if any, enhancements. 
Although Microsoft is no longer moving forward 
with the specific subset of LINQ that focuses on 
SQL Server and the integration with a graphical table 
designer, the core of LINQ to SQL is still supported. 
In this article I review the core features of LINQ to 
SQL and discuss some of the new roles of LINQ in 
the typical application architecture. I take a look at 
new LINQ features, such as Parallel LINQ and the 
use of LINQ in SharePoint. 


LINQ Capabilities 

Although Microsoft isn’t moving forward with the 
portion of LINQ that generates T-SQL code to 
run against SQL Server, the company still supports 
LINQ for processing query results. LINQ tends to 
be referred to in the context of a second technology 
group—the provider—such as LINQ to XML, LINQ 
to SQL, or LINQ to Entities. Each of these categories 
builds on the core LINQ syntax, and the differences 
between categories (e.g., LINQ to DataSets versus 
LINQ to SQL) can be subtle. A developer might 
initially suggest using LINQ to SQL even if he or 
she actually means LINQ to Entities or LINQ to 
DataSets. Some developers will continue leveraging 
LINQ to SQL as a familiar tool in certain situations, 
although LINQ’s larger focus is on data already 
returned from SQL Server. The area in which LINQ 
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continues to have a robust role is in processing data 
in the form of entities and the results from processing 
stored procedures against a database. 

As Figure | shows, LINQ provides six standard 
methods for accessing data (through Objects, Entities, 
SQL, DataSets, CAML, and XML). Of course there 
are literally dozens of third-party providers for LINQ, 
which is one of the strengths of LINQ as a technol- 
ogy. For SQL access, Microsoft provides LINQ to 
SQL. Microsoft also supports other technologies 
such as straight ADO.NET, the Entity Framework, 
and WCF data services for database access. Although 
most of these technologies have unique capabilities, 
several of them also overlap at some level. For exam- 
ple, the capabilities of LINQ to SQL are essentially 
a subset of the Entity Framework 4.0 capabilities. 
However, the Entity Framework was designed from 
the ground up to provide a more abstract layer than 
what's provided by LINQ to SQL. The Entity Frame- 
work abstraction adds some complexity but with the 
benefit of greater adaptability. 

Each of these data access technologies has dif- 
ferent characteristics that make its use appropriate. 
Some of those characteristics are technical—for 
example, LINQ to SQL supports only SQL Server 
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databases; if you need to query an Oracle database, 
LINQ to SQL isn’t an option. Others depend on 
application characteristics—for example, if you want 
to create abstractions of the core tables (i.e., convert 
the relational data into an object model), the Entity 
Framework is much more powerful than the other 
options, particularly LINQ to SQL. 


LINQ to SQL 

The logical starting point for getting into the nuts 
and bolts of LINQ and SQL Server is the core LINQ 
to SQL capability. Unlike old database technologies 
(e.g., Data Access Objects—DAOs), although Micro- 
soft hasn’t been developing enhancements for LINQ 
to SQL, the company hasn’t completely abandoned 
it. For example, the version of LINQ that shipped 
with .NET Framework 4.0 includes approximately 
50 changes, with a couple of them being potentially 
ground-breaking updates. (For more information, see 
Damien Guard’s blog post “LINQ to SQL changes in 
.NET 4.0” at damieng.com/blog/2009/06/01/linq-to- 
sql-changes-in-net-40.) However, if you read the list, 
you'll see that most of the enhancements focus on 
addressing issues rather than improving the tool or 
adding new data type support. As Guard notes in his 
blog, Microsoft’s official policy is that the company 
will improve the core of LINQ and might implement 
some customer requests, but the Entity Framework is 
the company’s primary focus. 

Clarifying the true core LINQ to SQL technologies 
is important. LINQ to SQL has essentially two execu- 
tion models. One such model is its support for the 
dynamic creation and submission of T-SQL queries 
to SQL Server. The second model is the execution of 
existing T-SQL such as stored procedures from which 
it returns a data set. The generation and execution of 
T-SQL commands is the heart of what's unique and 
might be better described as LINO to SOL Server. 
The logic used is specific to SQL Server as opposed 
to any other database management system (DBMS) 
and is part of the reason LINQ to SQL (unlike the 
Entity Framework) is SQL Server-specific. However, 
the core of this logic is really tied to the generation of 
dynamic SQL and was considered one of the initial 
strengths of LINQ. 

LINQ to SQL’s unique element isn’t its use of 
a data context, but rather what LINQ is doing as 
part of a dynamic query. For example, when you 
query a SQL Server data table with a LINQ query, 
LINQ actually generates T-SQL and submits that 
query to the database. However, if you instead use 
LINQ against the same context to execute a stored 
procedure, there’s no T-SQL to generate. Instead, 
your query will process the return from the stored 
procedure—a data set. Thus, although most people 
associate the use of stored procedures with LINQ to 
SQL and we talk about it and implement it in that 


context, the results of a query using a stored proce- 
dure could just as easily be run from an Entity Frame- 
work context or even from straight ADO.NET. The 
code is actually using LINQ to Datasets as opposed 
to LINQ to SQL. 

Because the DataContext object (eg., System 
.Data.Ling.DataContext) lets you reference a stored 
procedure, the initial and functional impression is 
that LINQ supports stored procedures. However, just 
using LINQ to execute a stored procedure is overkill. 
Some application developers might have only stored 
procedure access to their databases, in which case 
LINQ to SQL probably isn’t the best data access 
technology. You would probably leverage the Entity 
Framework in a .NET Framework 4.0 application, 
using the Entity Framework for other tasks. 

Another scenario you might face is that you're 
using .NET Framework 3.5 or 3.0 but aren’t yet on 
.NET Framework 4.0. In that case the Entity Frame- 
work is less than optimal because although previous 
versions support the Entity Framework, it isn’t really 
a viable choice until .NET Framework 4.0. You might 
opt to work with LINQ in such a case, or you might 
prefer to work with ADO.NET and data sets instead, 
leveraging LINQ in the middle tier. 

On the other side, there’s the question of moving 
forward. What if you’ve already invested heavily in 
LINQ—are you stranded? The answer is no. When 
you're ready to move forward, you should migrate to 
the Entity Framework and create a new data context 
based on your entities. As long as you have entities 
that match the tables you used with LINQ to SQL, 
you can simply change the data context for your 
LINQ query. Although the process isn’t automatic, 
taken in the larger context of moving to the Entity 
Framework the transition isn’t too painful and lets 
you continue to leverage the effort spent creating your 
LINQ to SQL queries. 


Parallel LINQ (PLINQ) 

Parallel extensions for LINQ are a set of commands 
that can be used to let LINQ process large quantities 
of data in parallel. A comprehensive discussion of 
PLINQ is beyond the scope of this article, but it’s 
important to address where PLINQ intersects with 
LINQ to SQL. For example, you might wonder 
whether parallel extensions let LINQ generate mul- 
tiple queries against SQL Server. The short answer 
is no. 

PLINQ queries are a set of extensions you can 
add to any LINQ query, even LINQ to SQL queries. 
However, PLINQ targets LINQ objects in memory. 
Thus, adding PLINQ syntax to a LINQ to SQL 
query has no beneficial effect. Keep in mind that 
PLINQ is optimized for LINQ to Objects and LINQ 
to XML, which are versions of LINQ that are wholly 
in memory. Although you might see a benefit with 
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a different provider, these providers have been opti- 
mized for PLINQ. 

As with any tool that involves parallel processing, 
PLINQ should be used with care. If you're expecting 
results in a specific order, note that using a parallel 
query can change the order. PLINQ can even change 
which results are returned if you combine it with a 
filter such as TOP. Use PLINQ extensions with care 
on existing queries. 

Adding the AsParallel option tells the processor to 
try to run the query in parallel—but doing so requires 
a certain amount of processing. If the amount of 
processing to break a query across multiple threads 
is greater than the time needed to process the query, 
performance decreases. Note that the parallel engine 
attempts to detect this scenario and will run a query 
sequentially if it detects that there would be little or 
no performance gain. However, this solution still isn’t 
perfect and does have an associated cost. 

To make a query run in parallel, simply call 
.AsParallel() when you create the query. 


var orderList = from order in 
currentDataContext.AsParallelQ 
where order.value > 100 
select order; 


The .AsParallel() extension includes a series of 
operators that let you further define how the query 
should run. For example, using the .WithDegreeOf 
Parallelism(#) operator instructs the library to limit 
the number of threads to the number provided. 


var orderList = from order in 
currentDataContext.AsParallelQ 
.WithDegreeOfParallelism(2) 
where order.value > 190 
select order; 


PLINQ extensions are targeted at providers such as 
LINQ to Objects and LINQ to XML that work in 
memory. They're implemented as extension methods 
and as such are available with any LINQ provider. 
However, because PLINQ extensions don’t interact 
within the core of the LINQ provider, they don’t 
provide custom capabilities for a provider such as 
LINQ to SQL. PLINQ spreads the processing of 
in-memory query execution across multiple threads. 
For more information about PLINQ, see Microsoft’s 
Introduction to PLINQ page at msdn.microsoft.com/ 
en-us/library/dd997425.aspx. 


LINQ in SharePoint 

Developers no longer need to learn Collaborative 
Application Markup Language (CAML) to create 
custom data views in SharePoint 2010; they can now 
use LINQ. The underpinning of LINQ in SharePoint 
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is the ability to generate CAML queries using the 
LINQ syntax, making it much easier to work with 
custom SharePoint data. SharePoint’s use continues 
to spread within and across organizations. Given the 
fact that SharePoint’s Business Data Connectivity ser- 
vice integrates with SQL Server, you might soon have 
to integrate your data directly into SharePoint. 

SharePoint 2010 introduces the concepts of server- 
side and client-side code, which have different object 
models. In theory, you should work on the server-side 
code because it’s important to screen data before it’s 
sent to clients. 

Recommended solutions for using client-side code 
involve using the SharePoint client object model. This 
set of objects supports two sources for accessing data: 
the SPQuery object and several named collection 
objects within the object model. The first, SPQuery, 
lets you define a CAML query. Unfortunately, by 
definition this object’s use of CAML indicates that it 
doesn’t support the LINQ syntax. 

Most of the other objects within the client 
object model support the [Enumerable(Of T)—aka 
TEnumerable<T>—interface. Any object that sup- 
ports this interface can be part of a LINQ to Objects 
query. However, this capability isn’t new, and LINQ to 
Objects is outside the scope of this article. Although 
you can use LINQ against data in the SharePoint cli- 
ent object model, doing so is a function of the existing 
LINQ to Objects implementation. 

On the server, you can leverage the objects 
in the Microsoft.SharePoint.Linq namespace. This 
namespace implements a LINQ to CAML provider. 
The code runs on the server, which is where you need 
to screen data before displaying it. The core capabili- 
ties are similar to those of LINQ to SQL. Instead of 
using a data context that points at a database, you can 
use one that points to a URL within your SharePoint 
site. The return from this URL is a list, or more spe- 
cifically an instance of an EntityList<Of T> object. 
The EntityList object is a strongly typed generic class 
that represents a SharePoint list. 

To use the LINQ to CAML provider, you must 
create an instance of your data context, then call 
GetList<Of T> to get the EntityList<Of T>. After 
you have this list you can create standard queries 
against it and return the contents of the list, or even 
insert the contents into a different list. Listing 1 
contains the code to create a DataContext from a list 
of orders, which can then be queried for orders only 
from a specific store, which are then inserted into the 
context of the list shown to the store. 

As you can see in Listing 1, the context represents 
your SharePoint site. Each list can be retrieved by 
name, but you must know and define the type of 
object used in the list. In this case, because I have con- 
trol, I used the same type of object within my target 
list. However, if the Store Orders list had a unique 
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LISTING l: Code to Create a SharePoint DataContext 


// Get the context for the site 
Microsoft.SharePoint.Linq.DataContext sharePointDC = 
new Microsoft.SharePoint.Ling.DataContext (SPContext.Current.Web.Ur1); 


// Get the order list 


Microsoft.SharePoint.Ling.EntityList<Orders> allOrders = 
sharePointDC.GetList<Orders>("All Orders"); 


// Get the store's list 


Microsoft.SharePoint.Ling.EntityList<Orders> ordersPerStore = 
sharePointDC.GetList<Orders>("Store Orders"); 


// Query the list for the current store "storied" 
var storeOrders = from order in allOrders 
where order.StoreID == storeID 


select order; 


// Create the store's list 
foreach (var storeOrder in storeOrders) 


ordersPerStore. InsertOnSubmit(storeOrder) ; 


i 


sharePointDC. SubmitChanges() ; 
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object OrderDetail instead of Order, then within the 
foreach loop in the listing I could create a new object 
of that type, assign the appropriate values (perhaps 
omitting details the store shouldn’t see), and insert my 
new object into the ordersPerStore object. 

Obviously, working with SharePoint involves more 
detail in terms of setting up a list and understand- 
ing the underlying types. However, as a method for 
working with data in SharePoint, LINQ provides a 
much richer interface than trying to perform the same 
queries in CAML. 


Still a Viable Solution 
LINQ is a powerful tool for working with SQL 
Server data. Just because Microsoft currently has 


the core of LINQ to SQL primarily in maintenance 
mode doesn’t mean that you shouldn’t continue to 
use LINQ to SQL. More importantly, you can still 
expect to encounter the LINQ syntax even if it isn’t 
used in a LINQ to SQL context. LINQ to SQL is 
a still a viable tool for 1:1 mapping of tables and 
stored procedures within a SQL Server database. But 
don’t expect a future release to include the graphical 
designer or bidirectional database structure updates. 

LINQ, when used with relational data, has tran- 
sitioned from focusing on translating an application 
development syntax such as C# into dynamic param- 
eterized T-SQL. The Q in LINQ has always stood 
for Query—and working with query results is where 
LINQ continues to shine. The generation of create, 
update, and delete statements is better left to tools 
such as Entity Framework that can focus on abstract- 
ing the mapping of relational structures. 

The introduction of LINQ for processing data 
within SharePoint and as part of PLINQ shows its 
versatility. Although LINQ to SQL isn’t getting new 
features, experienced application developers can still 
take advantage of its original capabilities. None of 
the newer technologies truly make those core ele- 
ments obsolete. If you’re already using LINQ (as I 
have been for a while), there’s no reason to stop. If 
you need a particular functionality that isn’t part 
of the core LINQ to SQL package, you can always 
leverage stored procedures and the Entity Framework 
to manipulate your data store, while using LINQ to 
screen and manipulate the data sent to and from these 
existing tools. SOL} 
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searching by keyword, topic, or mastered search engine rankings but 
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industry change 
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No Budget for Travel? No Problem! 
Get the training you need right at your desk with 


eLearning Courses 


http://elearning.left-brain.com 


Join industry experts for informative eLearning courses. 
Each course includes in-depth sessions as well as live Q&A. 


Our eLearning Series provides you with in-depth training 
on a variety of topics ranging from: 


Windows 7 

SQL 

Visual Studio 
.NET 

SharePoint 

And Much More! 


Visit http://elearning.left-brain.com and view all our available 
classes. You can attend live or view a past course on-demand. 


Don’t miss this opportunity for the training you need from the 
comfort of your own computer. 


Check out the eLearning Series offerings s today! i 


PureCM Professional 
Manage project source code with ease 


hen it comes to software configuration man- 

agement (SCM), .NET developers have a wide 
array of choices ranging from free, open-source solu- 
tions to expensive, proprietary products. Choosing the 
right tool for the job can make the difference between 
hitting milestones and slipping past the intended ship 
date. Most SCM solutions differentiate themselves with 
exclusive features such as robust, scalable, distributed 
capabilities and highly secure change-management 
triggers. For .NET programmers, a key ingredient in 
whatever product is selected is its tight integration with 
the Visual Studio (VS) environment. Such integration 
must work seamlessly by making the check-in/check- 
out, merge, diff, and branching functions as intuitive 
as the standard Save As file option. Such a product 
shouldn’t get in the way of productivity by saddling 
the developer with unnecessary modal dialogs, menus, 
and nonstandard workflows. PureCM (www.purecm 
.com) is a cross-platform SCM solution that meets 
many of these criteria. 

PureCM is sold in two editions, Standard and 
Professional. The Standard edition supports server- 
centric tracked revisions, fast client file distribution 
via compression, atomic commits, CVS/SVN-style 
copy-modify-merge and rollback functions, pre/post 
commit triggers for autonomous workflow scripting 
needs, private multiuser workspaces for better code 
isolation, a three-way merge tool to resolve code 
conflicts, granular ACL-based user/group manage- 
ment with built-in Windows Active Directory (AD) 
support, and integration with VS, Eclipse, and some 
third-party products. 

The Professional edition adds automated merging 
between streams (aka branches) for facilitated parallel 
development, caching servers for distributed teams, 
and, most important, an integrated issue tracking and 
reporting solution. Although PureCM’s issue tracking 
capabilities aren’t as extensive as other commercial 
dedicated issue tracking products, the tight integra- 
tion with PureCM-managed code streams adds a 
valuable amount of efficiency to problem identifica- 
tion, code assignment, and issue resolution. Likewise, 
the reporting module is straightforward and contains 
pertinent information most project managers and 
developers need when assessing a project’s state 
in the overall application life cycle. For additional 
comparisons between the Standard and Professional 
editions, go to PureCM’s comparison page at www 
.purecm.com/compare_editions.php. 
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Because PureCM is a client/server-based SCM solu- 
tion, there’s a two-phase process to install each compo- 
nent. The server installation is painless and can be done 
on the same machine as the client. This configuration 
is useful for testing or evaluation. Client deployment 
is easy; the setup process automatically installs and 
configures VS integration if the IDE is already installed 
on the client machine. PureCM’s VS integration, which 
Figure | shows, provides source-control functional- 
ity within the IDE. Working with PureCM is very 
intuitive, especially for developers who have experience 
with other source-control systems. In addition to the 
easy GUI-based navigation, features such as group 
project folders make setting broad-stream permissions 
straightforward. Merges are easy as well. PureCM uses 
its own naming conventions for familiar operations. 
Fortunately, these terms make sense and are quickly 
accessible. Figure 2 shows PureCM’s Repository and 
Streams organizational elements. 

Third-party plug-in support is also available. Cur- 
rent support includes integration for ThoughtWorks’ 
CruiseControl.NET continuous integration server and 
Axosoft’s OnTime project management suite. In addi- 
tion, PureCM’s own deployment service allows non- 
PureCM clients to access PureCM server-hosted files. 

In addition to supporting the .NET platform, 
PureCM provides cross-platform functionality via its 
Java implementation and Eclipse plug-in offerings. 
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PureCM’s Visual Studio integration 
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Although not as attractive as its VS counterpart, the Eclipse plug- 
in is just as feature-rich and functional. For development shops 
that deliver both Java and .NET solutions, PureCM’s multisystem, 
multilanguage support minimizes administrative overhead and the 
hassles of dealing with a different SCM interface—and possibly 
even back-end administration—depending on the type of project 
being managed. 

To assist in the evaluation process, the PureCM website pro- 
vides a quick tour of the product (www.purecm.com/quicktour 
.php), several video demonstrations (www.purecm.com/videos 
.php), and online product documentation (www.purecm.com/ 
documentation.php). PureCM customers can also access the 
company’s online knowledge base and online support ticket and 
contacts at support.purecm.com. 

Of course, if a .NET application development team requires 
only a simple source-version-control system, free alternatives 
such as Subversion or Git are more than adequate. One of the 
criticisms against PureCM is its lack of import capability of 
more recent source-control systems, such as SVN or Git. One 
could argue that such an import is unnecessary because the 
files can simply be targeted for inclusion. Unfortunately, this 
approach negates any previous revisions. At least the product 
has import facilities for older version-control solutions, such as 
Visual SourceSafe, CVS, and Perforce. 

If broader functionality, regulatory adherence, and robust VS 
IDE integration are paramount, PureCM delivers the goods at 
a reasonable per-seat price ($499 per concurrent user). Interested 
developers can visit www.purecm.com/downloads.php to download 
the client and server installers for their target OS. The installation 
includes a 30-day trial license for five users. The setup also includes 
a demo project formatted for VS 2005, which also works with VS 
2008. Overall, developers looking for a commercially supported, 
inherently secure, easy-to-manage, and fair-market-priced source- 
control management solution should take a look at PureCM. EG 
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TUNING Editor’s Tip 
DBSophic Offers Workload Got t 
Tuning for SQL Server ot a grea 


new product? 
Send announce- 
ments to products @ 
sqlmag.com. 
— Brian Reinholz, 
editorial web architect 


DBSophic has released Qure for SQL 
Server, a new product that uses a new 
approach to database optimization 
called workload tuning. According to 
the vendor, unlike query-to-query tun- 
ing, workload tuning optimizes the 
entire database workload automatically. 
Since a workload may include millions 
of queries, the process runs offline, 
incurring no load on the production 
environment. A free trial version of 
Qure for SQL Server is available at 
www.dbsophic.com. 


DATABASE DEVELOPMENT 

WhereScape Enhances IDE for Data Warehousing 

WhereScape has released WhereScape RED 6.5, the latest version of its integrated development environment 
for data warehousing. The latest version includes a number of features to better support agile data warehous- 
ing, such as: automated generation of standard code; automated development of standard BI logic; integrated 
operations framework with job scheduler and error reporting; version management for stored procedures 
and database objects; the ability to load and back out changes 
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Nob Hill Software Enhances Data Profiler SI Conan sor Sad 
Nob Hill Software has updated Data Profiler, a monitoring v. Were = 
product that tracks changes to data. The product can take par SENEO 
snapshots of your data or run in the background to monitor J. s Aaen, 
changes on a group of tables and send email alerts concerning 1 o e Senin ainiin 
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supports MSSQL, MySQL, and any database supporting 
OLEDB. To learn more, visit www.nobhillsoft.com. 
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DATABASE ADMINISTRATION 


Incredibly useful tools for SQL Server people 
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Linked Server Copy, which lets DBAs copy linked servers from 
one SQL Server instance to another. Other features of Linked 
Server Copy are: view all the linked servers that exist on the 
source server, or copy linked servers from the source instance to 
a destination instance. To learn more about SQL admin toolset, 
visit www.idera.com. 
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QweryBuilder 6.2 Simplifies Querying 


Werysoft has released QweryBuilder 6.2, an update to 


its database development and querying tool. Qwery- 


Builder lets end users extract data without writing 
code. The product also helps developers more easily 


ee Ee e o create queries, procedures, tables, views, and triggers. 


Source Column. | Target Column 


CustomerID 


a =| | The primary improvement in version 6.2 is a new data 


NemeStyie 


«| | transfer utility that lets you move data between two 


Tite 


supported data sources. QweryBuilder also works with 


ASE, SQL Anywhere, Oracle, and ODBC databases. 
To learn more, visit www.werysoft.com. 


E meme SECURITY 


Hedgehog 4.0 Enhances Database 
Scanning 

Sentrigo has released Hedgehog Enterprise 4.0, the 
latest version of its database monitoring solution. The 
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called Hedgehog DBscanner, a vulnerability assess- 
ment and scanning solution that integrates security 
and compliance events with most enterprise network 
and security monitoring systems. DBscanner conducts 
3,000 different checks of your databases, pinpoints 
specific tables containing restricted information, checks 
for password vulnerabilities, provides regulatory com- 
pliance report templates, and more. To learn more, visit 
www.sentrigo.com. 


SECURITY 

ScriptLogic Security Explorer Enhances 
Compliance 

ScriptLogic has announced Security Explorer 7.5, 
the latest version of its product for real-time manage- 
ment of access controls and security on workstations. 
Security Explorer helps IT pros apply security policies 
consistently across an organization’s servers. One of 
the most significant enhancements to version 7.5 is that it works with the Exchange 2010 Role Based Access 
Control (RBAC) permissions model, letting IT use RBAC to streamline assigning roles. To learn more or 
download a free 30-day evaluation, visit www.scriptlogic.com. 


STORAGE 

Red Gate Updates SQL Storage Compress 

Red Gate Software has released SQL Storage Compress 5.0, the latest update to its product that reduces the 
storage footprint of SQL Server databases. Enhancements to the latest version include: compression of up to 
90 percent; the ability to run compressed, live databases; transaction log backups created with SQL Server, 
SQL HyperBac, SQL Backup Pro, and other third-party backup products; performance improvements up to 
25 percent; and new support documentation. To learn more or download a trial, visit www.red-gate.com. 


DATABASE ADMINISTRATION 

CozyRoc Releases SSIS+ 1.5 Library 

CozyRoc has released CozyRoc SSIS+ 1.5, a library of 48 third-party components for SQL Server Integration 
Services for secure communications, compression, encryption, script reuse, and EDI data processing. New 
features in the latest version include: email support; Excel support; and integration with Dymanics CRM, 
SharePoint, and Parallel Loop. The product also integrates with IBM DB2, Oracle, SalesForce, and Amazon. 
To learn more or download the trial version, visit www.cozyroc.com. SQL 
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SP.NET applications have become the standard 

for most business applications. Their browser- 
based nature lets you roll out new applications with- 
out touching any client systems. ASP.NET is such a 
feature-rich programming platform that it’s easy to 
overlook some of its basic features. Here are seven 
ASP.NET coding techniques. 


How to get your connection 
string 

When developing ASP.NET applications, you can 
choose where to store your application’s connection 
string. The best place is almost always in the web 
.config file. The web.config file is in XML format and 
the appSettings key can hold multiple connections 
strings. Below, I use the appSetting key to store con- 
nection information: 


<appSettings> 

<add key="" MyConnectionString 
value="Server=MyServer;uid=My 
Username; pwd=MyPassword; 
database=MyDB" /> 

</appSettings> 


To use the values from the web.config file in your 
applications, include the System.Configuration 
namespace in your code and use the Configuration- 
Manager object: 


SqIConnectioncn = new Sql 
Connection(Confi guration 
Manager.ConnectionStrings 
["MyConnectionString"] 

.ConnectionString) ; 


How to turn off the display of 
non-visual controls 

When you're using data binding or many non-visual 
Ajax controls, the designer can get cluttered. To toggle 
off the display of ASP.NET non-visual controls, use 
Ctrl+Shift+N. 


How to display a blank item on 
the DropDownList 

The DropDownList control lets users select friendly 
display values. But by default, when the control is 


Fag 7 Easy ASP.NET Techniques 


databound it displays the value of the first item from 
the database. To display a blank item instead, add a 
value to the Items collection with an empty string in 
the Text property. Then set the AppendDataBound- 
Items property of the DropDopwnList to True. This 
inserts the blank value into the DropDownList when 
it’s initially displayed. 


How to set a web page’s 
default button 

Sometimes you need a specific button to handle the 
events when a user presses Enter. To set the default 
button on a web page, use the form’s DefaultButton 
property. Here I set a default button, Button], for 
forml: 


<form id="form1" DefaultButton= 
"Button1" runat="server"> 
</form> 


How to set the default focus to 
a control 

Another handy technique is setting a given control on 
a web page to have the focus when a user first opens 
the page. Using the form’s DefaultFocus property, I 
set the focus to a TextBox control named TextBox1: 


<form id="form1" DefaultFocus= 
"TextBox1" runat="server"> 
</form> 


How to highlight validation 
errors 

After performing input validation, you might want to 
position the cursor to the input control that generated 
the validation error, highlighting the values in that 
control. Using the document object’s select method, 
I select and highlight the contents of the TextBox 
named TextBox]: 


document .getElementById('Text 
Box1').selectQ; 


How to get rich UI controls 

CodePlex has a rich set of 40 Ajax controls available 

for free (ajaxcontroltoolkit.codeplex.com). SQL 
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SOL Sentry products are made by DBAs, for DBAs. They provide unparalleled insight, awareness and control over your 
SOL Server environment. With features like real-time and historical performance analysis and visual schedule 
management, SOL Sentry is the ultimate visibility solution. 
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